Problem Matcher for testing/CI

[zxlin]

How can we enable problem matchers for various testing tools like https://github.com/mochajs/mocha? Is there a reporting format that we should use for it to be matched by the runner when errors occur in the test cases? I saw the problem matchers for eslint like estlint-stylish but didn't see anything else.

If anyone can point me in the right direction, I'm happy to create the matcher if it doesn't already exist.

[bryanmacfarlane]

We've gone back and forth on this. We considered adding problem matchers directly for mocha, jest, eslint etc. but that's a growing list and it would be hard to remove.

We also discussed creating other setup actions like setup-jest etc. and also inputs to this but that's cumbersome to the consumers in their yaml.

I think the best direction would be if this action detected a framework (jest is in the package.json etc. etc.) and dynamically registered the problem matcher. There's a :: stdout command to register a problem matcher on the fly. We could do the same for typescript and eslint without hard coding it.

I'm also about to start a v2-beta so sketching this out could be part of that.

[zxlin]

@bryanmacfarlane Thanks for the reply, I agree this is a good idea, using package.json would be good for detection, I just wonder if people have multiple testing suites installed, perhaps for frontend vs backend services, would there be an issue/confusion.

It does look like eslint is already supported with their "stylish" reporter, I've been using that very successfully with Actions.

FYI on relevant thread over at mocha: mochajs/mocha#4132

It doesn't seem like the maintainers of mocha are particularly interested in a dedicated reporter, so we'll either need to create a new reporter or conform with one of their built-in reporters to support this.

I'm very happy to contribute to the matcher here since we use Actions pretty heavily and the current scroll all the way to bottom for test case errors is slightly inconvenient 😅