[JUnit Platform] Use EngineDiscoveryRequestResolver

[mpkorstanje]

🤔 What's changed?

Cucumber now uses JUnits EngineDiscoveryRequestResolver. JUnit provides a EngineDiscoveryRequestResolver that handles the resolution of most resources. This removes some complexity from Cucumber. This comes with several implications.

Classpath resources must be singular feature files.

There is now a distinction between the classpath resource and package discovery selectors. The former should be used for individual feature files on the classpath, the latter for directories.

So instead of using a class path resource selector for a directory:

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("com/example")
public class RunCucumberTest {
}

The package selector should be used instead:

@Suite
@IncludeEngines("cucumber")
@SelectPackages("com.example")
public class RunCucumberTest {
}

This is still okay:

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("com/example/my.feature")
public class RunCucumberTest {
}

Using the classpath resource selector on a folder does still work, but the functionality has been deprecated will result in a warning.

Support cucumber.execution.order

Features and scenarios are now discovered in arbitrary order and ordered after discovery. This order can be controlled with the cucumber.execution.order property. By default features are executed in lexical uri order, with scenarios and examples from top to bottom. Other valid options are reverse and random.

There is room to add custom orderer here but this hasn't been implemented yet.

Use JUnits discovery issue reporting mechanism

Any problems with the build are now reported through JUnits discovery issue reporting mechanism. This provides a more robust report than logging warnings.

⚡️ What's your motivation?

• Utilize Support scanning for classpath resources junit-team/junit-framework#3705 and Add resource container selector resolver junit-team/junit-framework#3718
• Adopt JUnit 5.13's DiscoveryIssue-reporting mechanism #2987
• Enable Random order in cucumber engine #2849

🏷️ What kind of change is this?

• 🏦 Refactoring/debt/DX (improvement to code design, tooling, documentation etc. without changing behaviour)

♻️ Anything particular you want feedback on?

📋 Checklist:

• I agree to respect and uphold the Cucumber Community Code of Conduct
• I've changed the behaviour of the code
  • I have added/updated tests to cover my changes.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly.
• Users should know about my change
  • I have added an entry to the "Unreleased" section of the CHANGELOG, linking to this pull request.

[codecov]

Codecov Report

Attention: Patch coverage is 91.81034% with 19 lines in your changes missing coverage. Please review.

Project coverage is 84.86%. Comparing base (6549826) to head (b6ab3cb).
Report is 33 commits behind head on main.

❗ Current head b6ab3cb differs from pull request most recent head 294d880

Please upload reports for the commit 294d880 to get more accurate results.

Files	Patch %	Lines
...it/platform/engine/CucumberDiscoverySelectors.java	81.25%	9 Missing and 3 partials ⚠️
...ucumber/junit/platform/engine/FeatureResolver.java	95.65%	2 Missing and 3 partials ⚠️
...junit/platform/engine/PackageFilteringVisitor.java	81.81%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2835      +/-   ##
============================================
+ Coverage     84.80%   84.86%   +0.06%     
+ Complexity     2728     2719       -9     
============================================
  Files           331      334       +3     
  Lines          9428     9569     +141     
  Branches        902      932      +30     
============================================
+ Hits           7995     8121     +126     
- Misses         1111     1118       +7     
- Partials        322      330       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mpkorstanje]

The current implementation of the JUnit Platform Engine treats @SelectClasspathResource aimed at directories as if all resources in that directory should be considered. This is not the intended usage, for this @SelectPackages should be used.

Cucumbers documentation consistently uses @SelectClasspathResource.

This will become a problem if/when we start to use the EngineDiscoveryRequestResolver with (#2835, junit-team/junit-framework#3705, junit-team/junit-framework#3718) because @SelectClasspathResource will cause an exception to be thrown when used with a non-existing resource.

[marcphilipp]

This should also be interesting for you (if/when it gets merged):
gradle/gradle#29535

[marcphilipp]

Great to see this being used here now! 👍