Skip to content

Await dynamic tests when a node's execution fails - #6142

Merged
mpkorstanje merged 8 commits into
junit-team:mainfrom
IWillBurn:fix/parallel-await-dynamic-tests-on-failure
Oct 9, 2026
Merged

mpkorstanje merged 8 commits into
junit-team:mainfrom
IWillBurn:fix/parallel-await-dynamic-tests-on-failure

Conversation

@IWillBurn

Copy link
Copy Markdown
Contributor

When Node.execute() throws after it has already submitted dynamic tests,
NodeTestTask skips DynamicTestExecutor.awaitFinished(). With parallel execution, the
node's after() behavior runs and the node is reported as finished while its dynamic
tests are still pending, so they finish after their parent or even after the test plan
(see #6141 for a reproducer). In Jupiter, this affects a @ParameterizedTest, a
@ParameterizedClass, or a DynamicContainer whose stream throws. Similarly, the
@AfterEach methods of a @TestFactory whose stream throws run while its dynamic tests
are still executing.

  • NodeTestTask now awaits the submitted dynamic tests regardless of whether
    execute() threw. The original exception remains the primary failure.
  • TestFactoryTestDescriptor awaits the provided dynamic tests before the after-each
    callbacks and methods, even if the stream threw. The Node#execute Javadoc now
    documents this guarantee.

The new tests in HierarchicalTestExecutorTests and ParallelExecutionIntegrationTests
fail on main with both executor service implementations. The release notes entries
are in 6.2.0-M1; I'm happy to move them to a 6.1.x patch release if you'd like to
backport the fix.

Closes #6141


I hereby agree to the terms of the JUnit Contributor License Agreement.


Definition of Done

When a node's execute() method threw an exception after submitting
dynamic tests to its DynamicTestExecutor, NodeTestTask skipped
awaitFinished() and executed the node's after behavior and reported it
as finished while the dynamic tests were still pending. With parallel
execution, these dynamic tests then finished after their parent, or
even after the end of the test plan.

In Jupiter, this happened when the stream of invocations of a test
template or class template (e.g. the arguments source of a
@ParameterizedTest or @ParameterizedClass) or the stream of children of
a DynamicContainer threw an exception.

NodeTestTask now awaits the submitted dynamic tests regardless of
whether execute() failed. The original exception remains the primary
failure.

Issue: junit-team#6141
Signed-off-by: IWillBurn <eugene.boyarnikov@gmail.com>
When the stream returned by a @testfactory method threw an exception
after providing some dynamic tests, TestFactoryTestDescriptor skipped
awaitFinished() and executed the after-each callbacks and methods of
the factory method while these dynamic tests were still executing in
parallel.

TestFactoryTestDescriptor now awaits the provided dynamic tests
regardless of whether the stream threw an exception, and Node#execute
documents that submitted dynamic tests are finished before the node's
after behavior, even if execute() throws.

Issue: junit-team#6141
Signed-off-by: IWillBurn <eugene.boyarnikov@gmail.com>
@mpkorstanje
mpkorstanje self-requested a review October 9, 2026 10:44
The dynamic tests now wait until the node's execute() method or the
stream has thrown, which establishes the order deterministically.

Issue: junit-team#6141
Signed-off-by: IWillBurn <eugene.boyarnikov@gmail.com>
The end-to-end tests could not reliably fail without the fix: a dynamic
test can only be released when the source fails, after which it races
with its parent. The unit tests release the dynamic test only once its
node either awaits it or calls after(), and verify that a test factory
awaits its dynamic tests when its stream fails.

Issue: junit-team#6141
Signed-off-by: IWillBurn <eugene.boyarnikov@gmail.com>
@IWillBurn

Copy link
Copy Markdown
Contributor Author

Follow-up to the latch review comment:

The latch-based tests passed CI, but replacing the sleeps with latches showed that the end-to-end tests can't reliably reproduce the bug, so I removed them. To show it, invocation a must still be running when @AfterAll starts. Releasing a in @AfterAll deadlocks with the fix (the parent waits for a), and releasing it when the arguments source fails leaves a race.

Instead, there are two deterministic tests:

  • HierarchicalTestExecutorTests: a node submits a dynamic test and throws from execute(). The dynamic test is released when the node either starts waiting for it (get() on its future) or calls after(). With the fix, it finishes before after(); without it, always after. This is the NodeTestTask path shared by @ParameterizedTest, @ParameterizedClass, and DynamicContainer, so it covers all of them.
  • TestFactoryTestDescriptorTests: a @TestFactory stream provides a dynamic test and then throws. The test verifies that awaitFinished() is called before the factory returns, i.e. before the after-each hooks. It's single-threaded.

I can add an end-to-end test back if you'd like, but as far as I understand, in Jupiter it can only be one of two: latch-based, which reliably passes with the fix but doesn't reliably fail without it, or time-based (sleep or timeout), which catches the bug but depends on timing.

@mpkorstanje

Copy link
Copy Markdown
Member

I can add an end-to-end test back if you'd like, but as far as I understand, in Jupiter it can only be one of two: latch-based, which reliably passes with the fix but doesn't reliably fail without it, or time-based (sleep or timeout), which catches the bug but depends on timing.

I don't think that is necessary. It is unfortunate that we lose the illustrative examples but I don't see any seams we can use to delay the execution of the dynamic tests.

@mpkorstanje mpkorstanje left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Cheers, excellent pull request.

@testlens-app

This comment has been minimized.

@marcphilipp

Copy link
Copy Markdown
Member

The release notes entries
are in 6.2.0-M1; I'm happy to move them to a 6.1.x patch release if you'd like to
backport the fix.

We plan to ship 6.2.0-M1 soon. Since this bug has been present for a long time I'd say we don't need to backport it at this time.

@mpkorstanje
mpkorstanje merged commit f1947c7 into junit-team:main Oct 9, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parameterized test invocations are abandoned when the arguments source throws during parallel execution

3 participants