ECONNRESET when running "many" tests

[schw4rzlicht]

Hey there,

we run into random ECONNRESET problems for weeks now and tried many things to mitigate the problem. We use supertest for e2e testing a GraphQL API built with Nest.js.

We run it like so:

GraphQLQuery.ts:

export class GraphQlQuery {
    private static _bindings: Map<string, supertest.SuperTest<supertest.Test>>;

    static queryGraphQl(app: INestApplication, query: GraphQlQuery, token?: string): supertest.Test {
        if (!GraphQlQuery._bindings) {
            GraphQlQuery._bindings = new Map();
        }

        let binding = GraphQlQuery._bindings.get(app.getHttpServer().testHash);
        if (!binding) {
            binding = request(app.getHttpServer());
            GraphQlQuery._bindings.set(app.getHttpServer().testHash, binding);
        }

        const req = binding.post("/graphql");

        if (token) {
            req.set({ Authorization: `Bearer ${token}` });
        }

        req.send({ operationName: null, query: query.build() })
            .expect((response) => {
                if (!response.ok) {
                    console.error(JSON.stringify(response, null, 2));
                }
            })
            .expect(200);

        return req;
    }

    // ...
}

Somewhere in the tests, we can then do

GraphQlQuery.queryGraphQl(app, query, jwt).then((response) => {
    expect(response).toBe(something);
});

The reason we do all the GraphQLQuery._bindings-stuff is because we are aware that those bindings should be reused. We create the Nest.js app in our beforeAll()-hooks, where we also add the testHash to the HTTP Server. It's basically a UUID to ensure that we can map bindings to test-suites. We also run Jest with --runInBand on the CI, so nothing should run concurrently. We have around 80 tests now which all rely on supertest and the more tests we build, the worse it gets. Up until to a point where we need to run the whole CI-step multiple times until ECONNRESET doesn't happen any more. Sometimes up to 10 times or so, with each pipeline needing ~5 minutes.

Unfortunately I didn't find much information about our problems and since Nest.js recommends supertest (and I think I can say that on behalf of the team, we really like the approach), I am wondering what we're missing.

Any help appreciated!

[jordancalder]

Any update here? I seem to be getting a similar error:

Error: ECONNRESET: Connection reset by peer
      at Test.assert (./node_modules/supertest/lib/test.js:193:18)
      at localAssert (./node_modules/supertest/lib/test.js:159:12)
      at ./node_modules/supertest/lib/test.js:156:5

[Dezzpil]

I wrote test to reproduce problem

describe('a lot of calls', function () {
  this.timeout(10000)

  it('should handle many requests', (done) => {
    const app = express();
    app.get('/', (req, res) => {
      res.send('hi');
    })

    const agent = request(app)

    const reqFn = (i) => {
      return new Promise((resolve, reject) => {
        const st = setTimeout(() => {
          clearTimeout(st)
          const req = agent.get('/')
          req.end((err, res) => {
            if (err) {
              console.log(i, err.message);
              reject(err)
            } else {
              res.text.should.be.eql('hi');
              resolve()
            }
          });
        }, 100)
      })
    }
    const promises = []
    for (let i = 0; i < 200; i++) {
      promises.push(reqFn(i))
    }
    Promise.all(promises)
      .then(() => done())
      .catch(err => done(err))
  });
});

There are may be different output in terminal after executions:

...
160 ECONNREFUSED: Connection refused
159 ECONNREFUSED: Connection refused

or

...
133 ECONNREFUSED: Connection refused
132 ECONNREFUSED: Connection refused

etc.

But i have no idea how to fix this :(

[schw4rzlicht]

@Dezzpil thanks a lot! But I'm not quite sure if that directly relates to our problem, since we only see ECONNRESET and never ECONNREFUSED :/

[lucasraziel]

same error with me. ECONNRESET is showing when running many tests, but It does not show when running the same test individually. In my case, I am not using Nest

[cmwd]

It seems that when calling supertest many times in a concurrent manner this code https://github.com/visionmedia/supertest/blob/28116f9060c1904a433e7d69f1c14db32cf60279/lib/test.js#L61
will spin up multiple HTTP servers or start listening on multiple ports (?).

My solution is to call app.getHttpServer().listen(0); after test app initialization in the beforeAll hook. This way the HTTP server can warm up and all supertest calls use the same address.

So far worked fine for me. Hope that helps in your case as well.

[niftylettuce]

Another issue that @shadowgate15 @spence-s and I found was that port 5000 is being used by AirPlay receiver on macOS Monterey via https://developer.apple.com/forums/thread/682332. I believe that @shadowgate15 is changing tests to use get-port instead to launch port for ZUUL / server.listen.

[simplenotezy]

Good find @niftylettuce - that might be it.

Perhaps we should find a solution to avoid using port 5000, if that is the case?

[simplenotezy]

@niftylettuce I did try to disable AirPlay Receiver, but still getting random "Connection reset by peer"

[shadowgate15]

We need to fix the tests to use beforeEach and afterEach to make sure the app is started on a random port and closes the app after the test.

This issue should be reopened @niftylettuce.

[shadowgate15]

And assign it to me I'll fix it later on this week.

[simplenotezy]

@shadowgate15 I believe the ticket was never reopened; however, still a relevant issue to tackle! 😊

[titanism]

PR welcome