Skip to content

chore: fix system-test-firefox screenshots_spec flake #20268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 18, 2022
Merged

chore: fix system-test-firefox screenshots_spec flake #20268

merged 1 commit into from
Feb 18, 2022

Conversation

ryanthemanuel
Copy link
Collaborator

@ryanthemanuel ryanthemanuel commented Feb 18, 2022 •
edited
Loading

User facing changelog

n/a

Additional details

There is logic in our screenshots that, if the image is big enough, and thus multi-part, we compare each part to the previous part to make sure that we are truly getting new data and not just the previous data. If the parts are the same, we retry until they are different or a timeout is reached. In some of the system tests, we are taking pictures of pages that are purely one color. In the tests I've changed, we don't limit the size of the screenshot to the viewport, so we are receiving huge images in 10 parts that are all exactly the same. This triggers the retry logic.

In the two tests I've updated, we do this 4 times and over time this is pressing on memory consumption which may cause firefox to crash. I'm fixing the tests as a part of this PR by limiting the screenshots to the viewport, but I think it's worth investigating if there's a less memory-intensive way of doing the retries (though this scenario may not occur too much outside of tests like this). I logged #20269 for this investigation

How has the user experience changed?

RAM Usage Before

image

RAM Usage After

image

PR Tasks

  • Have tests been added/updated?
  • [na] Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
  • [na] Has a PR for user-facing changes been opened in cypress-documentation?
  • [na] Have API changes been updated in the type definitions?
  • [na] Have new configuration options been added to the cypress.schema.json?

@cypress-bot
Copy link
Contributor

cypress-bot bot commented Feb 18, 2022

Thanks for taking the time to open a PR!

@ryanthemanuel ryanthemanuel marked this pull request as ready for review February 18, 2022 17:17
@cypress
Copy link

cypress bot commented Feb 18, 2022


Test summary

19278 • 0 • 218 • 0 • Flakiness 3

Run details
Project cypress
Status Passed
Commit 65172eb
Started Feb 18, 2022 5:07 PM
Ended Feb 18, 2022 5:18 PM
Duration 11:34 💡
OS Linux Debian - 10.10
Browser Multiple

View run in Cypress Dashboard ➡️

Flakiness

settings_spec.js 1 Flakiness
1 Settings > file preference panel > loads preferred editor, available editors and shows spinner
commands/net_stubbing_spec.ts 1 Flakiness
1 network stubbing > waiting and aliasing > can timeout waiting on a single request using "alias.request"
commands/navigation_spec.js 1 Flakiness
1 src/cy/commands/navigation > #visit > window immediately resolves and doesn't reload when visiting the same URL with hashes

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@ryanthemanuel ryanthemanuel mentioned this pull request Feb 18, 2022
Closed
@ryanthemanuel ryanthemanuel merged commit 75c8750 into develop Feb 18, 2022
@ryanthemanuel ryanthemanuel deleted the ryanm/chore/flake-system-tests-firefox branch February 18, 2022 18:54
tgriesser added a commit that referenced this pull request Feb 22, 2022
@tgriesser
Merge branch 'develop' into 10.0-release
47b1f38 
* develop: (35 commits)
  fix(deps): update dependency url-parse to v1.5.6 [security] (#20270)
  chore: fix cache keys to include PLATFORM (#20279)
  chore: fix server performance flake (#20271)
  test(system-tests): support docker-based tests against built binary (#20250)
  chore: fix system-test-firefox screenshots_spec flake (#20268)
  chore(deps): update dependency fs-extra to v9 🌟 (#19939)
  fix: Wait for child process to be ready (#19792)
  fix: treat form-data bodies as binary (#20144)
  test: replace cypress-test-example-repos coverage + remove bump (#20186)
  fix(driver): update wrapErr to ignore number and boolean values (#20172)
  release 9.5.0 [skip ci]
  chore: Update Chrome (stable) to 98.0.4758.102 (#20192)
  chore: enable volar.takeOverMode
  Add span names, merge develop
  fix: Update `.type(' ')` to not emit clicks when the keyup event has been prevented (#20156)
  test: remove redundant "other projects" CI jobs (#20133)
  chore(driver): move cy.focused and cy.root into their own files (#20054)
  Move sending root event to own script
  chore: release @cypress/vue-v3.1.1
  chore: release @cypress/react-v5.12.3
  ...
mschile added a commit that referenced this pull request Feb 23, 2022
@mschile
Squashed commit of the following:
96d86c7 
commit fc7149e
Merge: 0e942ed 0143e13
Author: Chris Breiding <[email protected]>
Date:   Wed Feb 23 10:28:34 2022 -0500

    Merge branch 'develop' into feature-multidomain

commit 0e942ed
Author: Chris Breiding <[email protected]>
Date:   Wed Feb 23 10:28:03 2022 -0500

    chore: Refactor multi-domain communication lifecycle (#20247)

commit 0143e13
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date:   Mon Feb 21 23:01:35 2022 +0000

    fix(deps): update dependency url-parse to v1.5.6 [security] (#20270)

    Co-authored-by: Renovate Bot <[email protected]>

commit 1fb16b0
Author: Zach Bloomquist <[email protected]>
Date:   Sun Feb 20 16:22:08 2022 -0500

    chore: fix cache keys to include PLATFORM (#20279)

    * chore: fix cache keys to include PLATFORM

    * build this branch

    * try using platform_key

commit 65ea8f7
Author: Ryan Manuel <[email protected]>
Date:   Fri Feb 18 17:57:34 2022 -0600

    chore: fix server performance flake (#20271)

commit ad2f4de
Author: Zach Bloomquist <[email protected]>
Date:   Fri Feb 18 18:37:22 2022 -0500

    test(system-tests): support docker-based tests against built binary (#20250)

    Co-authored-by: Ryan Manuel <[email protected]>

commit 75c8750
Author: Ryan Manuel <[email protected]>
Date:   Fri Feb 18 12:54:26 2022 -0600

    chore: fix system-test-firefox screenshots_spec flake (#20268)

commit 8d28261
Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Date:   Fri Feb 18 10:33:27 2022 -0600

    chore(deps): update dependency fs-extra to v9 🌟 (#19939)

    Co-authored-by: Renovate Bot <[email protected]>
    Co-authored-by: Emily Rohrbough <[email protected]>
    Co-authored-by: David Munechika <[email protected]>

commit ad50062
Author: Johannes Ewald <[email protected]>
Date:   Thu Feb 17 19:21:08 2022 +0100

    fix: Wait for child process to be ready (#19792)

    Co-authored-by: Zach Bloomquist <[email protected]>
    Co-authored-by: Zach Bloomquist <[email protected]>

commit 0d3e645
Author: Marco Lauinger <[email protected]>
Date:   Thu Feb 17 19:20:33 2022 +0100

    fix: treat form-data bodies as binary (#20144)

commit 42b0fce
Author: Zach Bloomquist <[email protected]>
Date:   Wed Feb 16 13:53:04 2022 -0500

    test: replace cypress-test-example-repos coverage + remove bump (#20186)

commit e55974c
Merge: f84bac5 ce956de
Author: Blue F <[email protected]>
Date:   Wed Feb 16 09:47:06 2022 -0800

    Merge pull request #20079 from cypress-io/issue-19403-perf-reporter-changes

    chore: Performance reporter changes

commit f84bac5
Author: Ali Kireçligöl <[email protected]>
Date:   Wed Feb 16 20:22:57 2022 +0300

    fix(driver): update wrapErr to ignore number and boolean values (#20172)

    Co-authored-by: Emily Rohrbough <[email protected]>

commit 1e6f51a
Author: BlueWinds <[email protected]>
Date:   Tue Feb 15 10:18:49 2022 -0800

    release 9.5.0 [skip ci]

commit 507b96f
Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Date:   Tue Feb 15 06:35:08 2022 -0600

    chore: Update Chrome (stable) to 98.0.4758.102 (#20192)

    Co-authored-by: cypress-bot[bot] <2f0651858c6e38e0+cypress-bot[bot]@users.noreply.github.com>

commit e8d7120
Merge: dff45ca bb26e98
Author: BlueWinds <[email protected]>
Date:   Mon Feb 14 14:15:27 2022 -0800

    Merge remote-tracking branch 'origin/master' into develop

commit dff45ca
Merge: a3f0d63 2bad703
Author: Blue F <[email protected]>
Date:   Mon Feb 14 12:59:52 2022 -0800

    Merge pull request #20142 from cypress-io/9b967e06f5-master-into-develop

    chore: merge master (9b967e0) into develop

commit 2bad703
Merge: 9b967e0 a3f0d63
Author: Blue F <[email protected]>
Date:   Mon Feb 14 12:26:14 2022 -0800

    Merge branch 'develop' into 9b967e0-master-into-develop

commit ce956de
Author: BlueWinds <[email protected]>
Date:   Mon Feb 14 08:21:15 2022 -0800

    Add span names, merge develop

commit ac1faf4
Merge: 961e764 2c88f0c
Author: BlueWinds <[email protected]>
Date:   Mon Feb 14 08:19:17 2022 -0800

    Merge remote-tracking branch 'origin/develop' into issue-19403-perf-reporter-changes

commit 961e764
Author: BlueWinds <[email protected]>
Date:   Thu Feb 10 14:21:44 2022 -0800

    Move sending root event to own script

commit bb26e98
Author: semantic-release-bot <[email protected]>
Date:   Thu Feb 10 15:16:39 2022 -0500

    chore: release @cypress/vue-v3.1.1

    [skip ci]

commit 6a96ca5
Author: semantic-release-bot <[email protected]>
Date:   Thu Feb 10 15:16:23 2022 -0500

    chore: release @cypress/react-v5.12.3

    [skip ci]

commit 9b967e0
Author: Lachlan Miller <[email protected]>
Date:   Thu Feb 10 16:26:20 2022 +1000

    fix: set correct default when using react-scripts plugin (#20141)

commit e709184
Merge: e0bf811 97e6c14
Author: Barthélémy Ledoux <[email protected]>
Date:   Wed Feb 9 20:03:05 2022 -0600

    Merge pull request #20132 from cypress-io/elevatebart/trigger-vue-release

commit 97e6c14
Author: ElevateBart <[email protected]>
Date:   Wed Feb 9 17:52:38 2022 -0600

    fix: create a dummy commit to trigger release

commit fa0b68a
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 14:50:04 2022 -0800

    Fix path

commit f7c46fc
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 14:35:00 2022 -0800

    Refactor async data into more convenient helper

commit 57decf4
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 13:21:57 2022 -0800

    Include honeycomb key so we can send root event

commit 3cf4a2b
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 12:55:50 2022 -0800

    Reduce event duplication

commit 9d433c7
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 12:50:24 2022 -0800

    Send root honeycomb event even if node_modules cache already exists

commit 705262f
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 10:48:17 2022 -0800

    Fix

commit b9c0d5e
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 10:37:52 2022 -0800

    fix indentation

commit f694bd2
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 10:33:15 2022 -0800

    Another attempt to wrap all events into one trace

commit 5e96b5c
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 09:52:17 2022 -0800

    Attepmt to associate all circleci jobs for a given build into one trace

commit 980b951
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 09:34:59 2022 -0800

    Tweak honeycomb events further

commit b8f1b4f
Merge: 0d834bc 52ed6ed
Author: BlueWinds <[email protected]>
Date:   Tue Feb 8 09:12:20 2022 -0800

    Merge remote-tracking branch 'origin/develop' into issue-19403-perf-reporter-changes

commit 0d834bc
Author: BlueWinds <[email protected]>
Date:   Mon Feb 7 12:25:35 2022 -0800

    Add data to all events sent, not just top level

commit ac5bfd3
Author: BlueWinds <[email protected]>
Date:   Mon Feb 7 11:37:55 2022 -0800

    Use honeycomb tracing

commit 304d8bd
Author: BlueWinds <[email protected]>
Date:   Mon Feb 7 10:22:04 2022 -0800

    Make performance-reporter for system tests use get-next-version script
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jennifer-shehane jennifer-shehane jennifer-shehane approved these changes

@mjhenkes mjhenkes mjhenkes approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ryanthemanuel @jennifer-shehane @mjhenkes