Best practices for setting up replica sets with Jest and many test files

[pavlos163]

I have setup this structure:

jest.config.js

module.exports = {
  preset: "ts-jest",
  globalSetup: "<rootDir>/src/tests/jest.setup.js",
  globalTeardown: "<rootDir>/src/tests/jest.teardown.js",
  ...
};

jest.setup.js

const { MongoMemoryReplSet } = require("mongodb-memory-server");

// Load dev env variables
require("dotenv").config({ path: "dev.env" });

const MONGO_VERSION = "7.0.11";

module.exports = async () => {
  if (process.env.DATABASE_IN_MEMORY_ENABLED === "true") {
    const replicaSet = await MongoMemoryReplSet.create({
      instanceOpts: [
        {
          port: parseInt(process.env.DATABASE_IN_MEMORY_PORT)
        }
      ],
      binary: {
        version: MONGO_VERSION
      },
      replSet: { count: 4, storageEngine: "wiredTiger" }
    });

    // eslint-disable-next-line no-undef
    globalThis.__MONGOD__ = replicaSet.servers[0];
  }
};

jest.teardown.js

module.exports = async () => {
  if (process.env.DATABASE_IN_MEMORY_ENABLED === "true") {
    // eslint-disable-next-line no-undef
    await globalThis.__MONGOD__.stop();
  }
};

An example test file:

import { clearDb, closeDb, connectDb } from "../../tests/utils/db";

describe("SomeTestService", () => {
  beforeAll(async () => await connectDb("SomeTestService"));
  afterAll(async () => await closeDb());

  describe("someTest", () => {
    describe("when specific scenario happens", () => {
      beforeAll(async () => {
        ...
      });
      afterAll(async () => await clearDb());

      it("should do something", async () => {
        ...
      });
    });
  });
});

db.ts

import mongoose from "mongoose";

async function connectDb(moduleName: string): Promise<void> {
  let url: string;
  if (process.env.DATABASE_IN_MEMORY_ENABLED === "true") {
    url = `mongodb://${process.env.DATABASE_IN_MEMORY_IP}:${process.env.DATABASE_IN_MEMORY_PORT}/${moduleName}`;
  } else {
    url = `${process.env.DATABASE_URL}-${moduleName}`;
  }

  await mongoose.set("strictQuery", false).connect(url);
}

async function clearDb(): Promise<void> {
  const collections = Object.values(mongoose.connection.collections);

  const BATCH_SIZE = 5;

  for (let i = 0; i < collections.length; i += BATCH_SIZE) {
    const batch = collections.slice(i, i + BATCH_SIZE);

    await Promise.all(batch.map(async (collection) => collection.deleteMany({})));
  }
}

async function closeDb(): Promise<void> {
  await mongoose.connection.dropDatabase();
  await mongoose.connection.close();
}

export { connectDb, clearDb, closeDb };

Does this follow best practices on how to structure tests with MongoDB in-memory?

I basically setup MONGOD in jest.setup.js, then in each one of the test files I connect to the instance (with a different database name), then after each test file I clear & close DB.

I have many test files (more than 100) and I sometimes see timeouts when I connect to my DB (when mongoose.connect runs) so I was wondering whether with my current structure I might be overloading my memory server.

FYI I use --maxWorkers=4 when running tests.

[hasezoey]

It looks fine to me, just some notes:

instanceOpts: [
       {
         port: parseInt(process.env.DATABASE_IN_MEMORY_PORT)
       }
     ],

Just so you are aware, this only sets the port for the first instance. (each entry in the instanceOpts applies to a specific instance)

replSet: { count: 4

MongoDB recommends to run uneven number of replsets, so i would suggest to either go with 3 or 5.

Also know that more severs on a single machine might slow it down due to synchronization and likely working on the same data storage medium.

url = `mongodb://${process.env.DATABASE_IN_MEMORY_IP}:${process.env.DATABASE_IN_MEMORY_PORT}/${moduleName}`;

It would work, but might be unreliable, i would recommend just setting the process.env.DATABASE_URL in the setup and always using that in connect.
See process.env.MONGO_URI in Integration with Test Runners: jest

I have many test files (more than 100) and I sometimes see timeouts when I connect to my DB (when mongoose.connect runs) so I was wondering whether with my current structure I might be overloading my memory server.

With just the given examples, it should likely not overload, but this might depend on the work you actually do in the tests themself (read / write).
Maybe check:

• if there is a change when changing to uneven number of replsets
• that the database directories are on a fast medium (i recommend tmpfs or similar)

[pavlos163]

Thanks a lot for your quick reply.

Could you please elaborate on this?

that the database directories are on a fast medium (i recommend tmpfs or similar)

[hasezoey]

In different wording, i meant to verify (or change) the database storage path to be on a filesystem (and its backing storage like RAM/SDD/HDD) that is fast (like tmpfs). The default path on linux is /tmp, but can be explicitly checked via the debug logs.
If this does not solve your question, could you be more specific on what you want me to elaborate more on?

[pavlos163]

I made the changes suggested in your original answer here:

jest.setup.js

const { MongoMemoryReplSet } = require("mongodb-memory-server");

// Load dev env variables
require("dotenv").config({ path: "dev.env" });

const MONGO_VERSION = "7.0.11";

module.exports = async () => {
  if (process.env.DATABASE_IN_MEMORY_ENABLED === "true") {
    const replicaSet = await MongoMemoryReplSet.create({
      binary: {
        version: MONGO_VERSION
      },
      replSet: { count: 3, storageEngine: "wiredTiger" } // Updated this to be 3
    });

    // eslint-disable-next-line no-undef
    globalThis.__MONGOD__ = replicaSet;

    const uri = replicaSet.getUri();

    console.log(`jest.setup.js uri -> ${uri}`);

    process.env.IN_MEMORY_DATABASE_URL = uri.slice(0, uri.lastIndexOf("/")); // Set URI here
  }
};

db.ts

import mongoose from "mongoose";

async function connectDb(moduleName: string): Promise<void> {
  let url: string;
  if (process.env.DATABASE_IN_MEMORY_ENABLED === "true") {
    url = `${process.env.IN_MEMORY_DATABASE_URL}/${moduleName}`; // Use URI set by jest.setup.js
  }
  ...

As I continued to get timeouts, I also updated my test command to use --runInBand, to see whether multiple Jest workers caused an issue - even with runInBand, I still get timeouts.

1. I am now going to ssh into my test runner machine to see whether /tmp has been set as a /tmpfs filesystem type.
2. I am attaching logs from my most recent run with timeouts here (logs after the timeout have been removed, see thrown: "Exceeded timeout of 10000 ms for a hook for the test failed due to the timeout)

8_Test.txt

[pavlos163]

I just verified that /tmp is not set as tmpfs in my Github action runners.

Would the best practice here be to pass something like /run (which is tmpfs) as dbPath?

[hasezoey]

Would the best practice here be to pass something like /run (which is tmpfs) as dbPath?

Anywhere is fine as long as it has permissions and enough storage space (wiredTiger requires about at least 300mb, even if completely empty)
Maybe consider using /dev/shm/ instead, this directory basically exists anywhere on linux and is (to my knowledge) always tmpfs and has the proper permissions. (i only just found out about this)

I am attaching logs from my most recent run with timeouts here (logs after the timeout have been removed, see thrown: "Exceeded timeout of 10000 ms for a hook for the test failed due to the timeout)
8_Test.txt

From my quick glance at the logs, it seems like it is trying to drop some stuff (start 2025-04-17T11:08:12.555+00:00), but that stuff is still in use and at the end of the logs is still not dropped (at 2025-04-17T11:08:59.554+00:00), this is a difference of about ~47 seconds. (if i read that correctly; i looked at ident: internal-161--2813674390592196994)

I personally have not encountered this before, try to check:

• do you have any open transactions at the time of drop?
• do you re-use the same database names across different test files? (TransactionService?)
• are your databases with test data so big that dropping them may take a while?
• as a last resort, consider increasing the timeout for the drop collection call hook. (or dont drop the collections if your storage space can affort that)

PS: i just noticed the uri.slice is not correct for replsets, consider not using it or manually adding /?replicaSet=testset which would otherwise get cut-off, though i dont think it is strictly required.