Problem/Motivation

Follow-up from #3267124: Temporarily skip failing tests.

The following JavaScript test methods are currently skipped for random fails:

Layout Builder

  1. LayoutBuilderUiTest::assertHighlightNotExists.

Other

Proposed resolution

  1. File children to debug and un-skip each.
  2. Un-skip one method per issue and run the affected test 500x.
  3. Queue 8 test runs of 1000x against MySQL or MariaDB environments.
  4. Post a patch that un-skips the test without any attempted fix to get a baseline fail rate, and always test this patch at the same time as any proposed fix with the same environment.
  5. Skip other methods in the test if it becomes necessary due to out-of-scope fails.

Remaining tasks

  1. Add items to the above summary for::

Already fixed

  1. \Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest::testButton: #3268368: Robustify and restore \Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest::testButton
  2. LayoutBuilderDisableInteractionsTest::testFormsLinksDisabled(): #3268680: [random test failure] Restore and fix LayoutBuilderDisableInteractionsTest::testFormsLinksDisabled()
  3. #3272797: [random test failure] Restore LayoutBuilderTest::testConfigurableLayoutSections()
  4. QuickEditIntegrationTest::testArticleNode()#3268244: [random test failure] Un-skip and fix QuickEditIntegrationTest::testArticleNode()
  5. QuickEditIntegrationTest::testCustomBlock()#3267258: Remove Quick Edit support from editor.module
  6. AjaxBlockTest::testAddAjaxBlock() #3304371: Fix intermittent failure in AjaxBlockTest
  7. ContentPreviewToggleTest::testContentPreviewToggle(): #3268678: [random test failure] Restore ContentPreviewToggleTest::testContentPreviewToggle()
  8. SettingsTrayBlockFormTest::testEditModeEnableDisable(): #3304901: Fix intermittently failing Settings Tray Functional Javascript tests
  9. LayoutBuilderDisableInteractionsTest::assertContextualLinksClickable()#3353103: [no random test failure] Try to un-skip and fix LayoutBuilderDisableInteractionsTest::assertContextualLinksClickable in context of [#3353085]
  10. #3353179: [random test failure] Try to un-skip and fix LayoutBuilderTest::testLayoutBuilderUi in context of [#3353085]
  11. #3353153: [random test failure] Try to un-skip and fix LayoutBuilderNestedFormUiTest::testAddingFormBlocksToDefaults in context of [#3353085]
  12. #3353092: [random test failure] Try to un-skip and fix ContextualLinksTest in context of [#3353085]
  13. #3353167: [random test failure] Try to un-skip and fix LayoutBuilderNestedFormUiTest::testAddingFormBlocksToOverrides in context of [#3353085]
  14. BlockFormMessagesTest::testValidationMessage()#3353088: [random test failure] Try to un-skip and fix BlockFormMessagesTest::testValidationMessage() in context of [#3353085]
  15. #3351494: Skip Drupal\Tests\media\FunctionalJavascript\MediaSourceFileTest

Comments

catch created an issue. See original summary.

longwave’s picture

Title: [PP-1] Stop skipping some JavaScript tests when chrome driver is updated » Stop skipping some JavaScript tests when chrome driver is updated
Related issues: +#3267823: \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testCustomBlock(). is failing on latest chromedriver
longwave’s picture

Status: Active » Needs review
StatusFileSize
new6.03 KB

Status: Needs review » Needs work

The last submitted patch, 3: 3267247-3-revert-50x.patch, failed testing. View results

longwave’s picture

So of the five tests here, repeated 50x each:

  • \Drupal\Tests\layout_builder\FunctionalJavascript\AjaxBlockTest
    passed 50 times
  • \Drupal\Tests\layout_builder\FunctionalJavascript\ContentPreviewToggleTest
    passed 48 times, failed twice
  • \Drupal\Tests\layout_builder\FunctionalJavascript\LayoutBuilderDisableInteractionsTest
    passed 50 times
  • \Drupal\Tests\layout_builder\FunctionalJavascript\LayoutBuilderTest
    passed 50 times
  • \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest
    failed 43 times, passed 7 times
longwave’s picture

ContentPreviewToggleTest has two different stack traces for the two fails:

1) Drupal\Tests\layout_builder\FunctionalJavascript\ContentPreviewToggleTest::testContentPreviewToggle
Behat\Mink\Exception\ElementHtmlException: Element exists on the page.

/var/www/html/core/modules/layout_builder/tests/src/FunctionalJavascript/ContentPreviewToggleTest.php:138
/var/www/html/core/modules/layout_builder/tests/src/FunctionalJavascript/ContentPreviewToggleTest.php:88

vs

1) Drupal\Tests\layout_builder\FunctionalJavascript\ContentPreviewToggleTest::testContentPreviewToggle
Behat\Mink\Exception\ElementHtmlException: Element exists on the page.

/var/www/html/core/modules/layout_builder/tests/src/FunctionalJavascript/ContentPreviewToggleTest.php:138
/var/www/html/core/modules/layout_builder/tests/src/FunctionalJavascript/ContentPreviewToggleTest.php:94

QuickEditIntegrationTest mostly fails with:

1) Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testCustomBlock
Failed asserting that null is identical to 'Body'.

/var/www/html/core/modules/quickedit/tests/src/FunctionalJavascript/QuickEditJavascriptTestBase.php:179
/var/www/html/core/modules/quickedit/tests/src/FunctionalJavascript/QuickEditIntegrationTest.php:312

but once with

1) Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testArticleNode
JavaScript condition met:
Drupal.quickedit.collections.entities.get('node/1[0]').get('state') === 'closed'
Failed asserting that false is true.

/var/www/html/core/tests/Drupal/FunctionalJavascriptTests/WebDriverTestBase.php:158
/var/www/html/core/modules/quickedit/tests/src/FunctionalJavascript/QuickEditIntegrationTest.php:262
xjm’s picture

Status: Needs work » Needs review
StatusFileSize
new4.26 KB

Let's try a few more runs of the passing ones to prove that they're passing.

longwave’s picture

The majority of QuickEditIntegrationTest fails might be something to do with the hover state; the field name is only shown in the toolbar when the pointer is hovering over the field itself; otherwise only the entity title is shown. I am not sure where the pointer is placed in tests by default, nor do I see anything that explicitly places the pointer in these tests.

Having said that, the test does not appear to use any random factors so the result should (in theory) be deterministic? Maybe this could be a race condition though, where the hover event does not always fire?

xjm’s picture

@longwave Yeah a good 75% or more of our random fails in JS functional tests historically are due to race conditions and not using the correct APIs to wait for elements to be rendered.

Status: Needs review » Needs work

The last submitted patch, 7: tests-3267247-5-revert-passing-100x.patch, failed testing. View results

xjm’s picture

The other three are definitely still failing plenty:
https://dispatcher.drupalci.org/job/drupal_patches/117467/console

For the 400 total runs of each in #7:

  • AjaxBlockTest had only a 5% fail rate
  • LayoutBuilderDisableInteractionsTest had a 17% fail rate
  • LayoutBuilderTest had an 8% fail rate
xjm’s picture

I also think the numbers in #5 are incorrect; there are a total of 43 failures and 2 errors in the results of #3. Sorry, read it the wrong way around.

xjm’s picture

Issue tags: +Random test failure

 

xjm’s picture

xjm’s picture

Title: Stop skipping some JavaScript tests when chrome driver is updated » [meta] Fix and re-enable tests skipped for random failures
Category: Task » Plan
Priority: Normal » Critical
Issue summary: View changes
Status: Needs work » Active

Converting this to a meta.

xjm’s picture

Issue summary: View changes

 

xjm’s picture

Issue summary: View changes

 

xjm’s picture

 

lauriii’s picture

phenaproxima’s picture

phenaproxima’s picture

xjm’s picture

#3268680: [random test failure] Restore and fix LayoutBuilderDisableInteractionsTest::testFormsLinksDisabled() is in, so we can use the mechanism added in that issue and see if it can be applied to other Layout Builder tests (or anything else that tests the off-canvas area).

xjm’s picture

Issue summary: View changes

 

xjm’s picture

Issue summary: View changes

 

phenaproxima’s picture

Issue summary: View changes
phenaproxima’s picture

Issue summary: View changes
phenaproxima’s picture

Issue summary: View changes
phenaproxima’s picture

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

xjm’s picture

Issue summary: View changes

 

spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Adding additional skipped tests. Sorry, LB.

bnjmnm’s picture

Issue summary: View changes
bnjmnm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

 

xjm’s picture

Issue summary: View changes

 

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

xjm’s picture

Issue summary: View changes
catch’s picture

catch’s picture

catch’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
dww’s picture

Issue summary: View changes

Tried to add #3351596: Skip Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest + #3351597: [random test failure] Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest to the summary, but it was unclear if the list at the top are the follow-up issues to un-skip things, or the original issues where we started skipping (it's some of both).

I added a remaining tasks to start to track which ones still need child issues to get them un-skipped and working.

Maybe it'd be more clear if we had separate lists:

1. Tests we've had to skip
2. Issues to fix + stop skipping them
3. Issues like #2 that are already fixed and the test is active again (the current "Already fixed" list)

? Not sure the best way to track this plan and all its issues... 😅

Thanks/sorry,
-Derek

catch’s picture

The three lists idea is good. We don't have issues to unskip all of the tests that we've skipped which is why it's inconsistent at the moment.

Also before opening individual issues, we probably want to try to get #2856047: Avoid random failures in JavascriptTestBase when testing functionality in a dialog done in case that allows us to fix all the dialog tests, and maybe check the related offcanvas logic is used everywhere consistently too. That's a large number of the currently failing/skipped tests at the moment.

xjm’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

spokje’s picture

Issue summary: View changes

Moved Already fixed issues to the Already fixed section.
(Life can be so easy sometimes...)

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

catch’s picture

Status: Active » Fixed

All of the issues in the issue summary are fixed now, closing this out!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.