chore: Performance reporter changes #20079

BlueWinds · 2022-02-07T19:40:39Z

Closes Unification: Performance Testing #19403

User facing changelog

No user facing changes.

Additional details

Updates the performance-reporter to send honeycomb events for specs as well as individual tests, bundling this all together into honeycomb traces for easier reporting / analysis.

For an example of the new tracing, see https://ui.honeycomb.io/cypress/datasets/systemtest-performance/result/6g9856meLqn/trace/aTmoUYdQLEb?span=22f3e7f4-3153-4f55-b758-a285dc4ccf10.

How has the user experience changed?

PR Tasks

[N/A] Have tests been added/updated?
[N/A] Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
[N/A] Has a PR for user-facing changes been opened in cypress-documentation?
[N/A] Have API changes been updated in the type definitions?
[N/A] Have new configuration options been added to the cypress.schema.json?

cypress-bot · 2022-02-07T19:40:42Z

Thanks for taking the time to open a PR!

Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

cypress · 2022-02-07T19:54:38Z

Test summary

19258 • 0 • 218 • 0 • 0

Run details


Project	cypress
Status	Passed
Commit	`ce956de`
Started	Feb 14, 2022 4:26 PM
Ended	Feb 14, 2022 4:38 PM
Duration	11:47 💡
OS	Linux Debian - 10.10
Browser	Multiple

View run in Cypress Dashboard ➡️

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

…eporter-changes

BlueWinds · 2022-02-09T00:13:34Z

system-tests/lib/performance-reporter.js

+// system test tasks and build steps into a single span.
+if (require.main === module) {
+  addAsyncInfoAndSend(circleCiRootEvent).then(() => {
+    console.log(circleCiRootEvent.data)


This log left in intentionally, so it shows up in circleci and gives us a way (not an easy way, but at least a way) to see the traceid associated with a specific build.

flotwig

🆒 ❗

flotwig · 2022-02-10T21:55:30Z

system-tests/lib/performance-reporter.js

+// This file is executed once as a script at the beginning of the circleci build,
+// so that we can send the root event exactly once and associate all the various
+// system test tasks and build steps into a single span.
+if (require.main === module) {


nit: IMO it would be easier for new people coming in to debug this if this "script" was in system-tests/scripts and not just conditionally running if this module is main. Separation of the script and the lib is how most of our scripts are set up.

tgriesser

Very cool - I think the main thing that's missing, at least from the trace you linked, is a "name" identifier of each span... I think in this case it'd be ci_run, spec_execution, test_execution

Also, do you think there's a way we can get a durationMs added to the root span?

BlueWinds · 2022-02-14T16:18:39Z

Very cool - I think the main thing that's missing, at least from the trace you linked, is a "name" identifier of each span... I think in this case it'd be ci_run, spec_execution, test_execution

Adding.

Also, do you think there's a way we can get a durationMs added to the root span?

I suppose it would be possible - I could add a new circleci step which depends on all the system-test steps, and delay sending the root span until all the others are executed. It would need a way to pass the 'start' timestamp between circleci stages (probably by writing it to a file in the system-tests node_modules cache).

So "yes, but not trivially" - probably as a separate PR at a later date.

…eporter-changes

BlueWinds · 2022-02-16T17:47:51Z

Since Tim is out the rest of the week and I addressed his requested changes, went ahead and merged without waiting for re-approval.

commit fc7149e Merge: 0e942ed 0143e13 Author: Chris Breiding <[email protected]> Date: Wed Feb 23 10:28:34 2022 -0500 Merge branch 'develop' into feature-multidomain commit 0e942ed Author: Chris Breiding <[email protected]> Date: Wed Feb 23 10:28:03 2022 -0500 chore: Refactor multi-domain communication lifecycle (#20247) commit 0143e13 Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Date: Mon Feb 21 23:01:35 2022 +0000 fix(deps): update dependency url-parse to v1.5.6 [security] (#20270) Co-authored-by: Renovate Bot <[email protected]> commit 1fb16b0 Author: Zach Bloomquist <[email protected]> Date: Sun Feb 20 16:22:08 2022 -0500 chore: fix cache keys to include PLATFORM (#20279) * chore: fix cache keys to include PLATFORM * build this branch * try using platform_key commit 65ea8f7 Author: Ryan Manuel <[email protected]> Date: Fri Feb 18 17:57:34 2022 -0600 chore: fix server performance flake (#20271) commit ad2f4de Author: Zach Bloomquist <[email protected]> Date: Fri Feb 18 18:37:22 2022 -0500 test(system-tests): support docker-based tests against built binary (#20250) Co-authored-by: Ryan Manuel <[email protected]> commit 75c8750 Author: Ryan Manuel <[email protected]> Date: Fri Feb 18 12:54:26 2022 -0600 chore: fix system-test-firefox screenshots_spec flake (#20268) commit 8d28261 Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Date: Fri Feb 18 10:33:27 2022 -0600 chore(deps): update dependency fs-extra to v9 🌟 (#19939) Co-authored-by: Renovate Bot <[email protected]> Co-authored-by: Emily Rohrbough <[email protected]> Co-authored-by: David Munechika <[email protected]> commit ad50062 Author: Johannes Ewald <[email protected]> Date: Thu Feb 17 19:21:08 2022 +0100 fix: Wait for child process to be ready (#19792) Co-authored-by: Zach Bloomquist <[email protected]> Co-authored-by: Zach Bloomquist <[email protected]> commit 0d3e645 Author: Marco Lauinger <[email protected]> Date: Thu Feb 17 19:20:33 2022 +0100 fix: treat form-data bodies as binary (#20144) commit 42b0fce Author: Zach Bloomquist <[email protected]> Date: Wed Feb 16 13:53:04 2022 -0500 test: replace cypress-test-example-repos coverage + remove bump (#20186) commit e55974c Merge: f84bac5 ce956de Author: Blue F <[email protected]> Date: Wed Feb 16 09:47:06 2022 -0800 Merge pull request #20079 from cypress-io/issue-19403-perf-reporter-changes chore: Performance reporter changes commit f84bac5 Author: Ali Kireçligöl <[email protected]> Date: Wed Feb 16 20:22:57 2022 +0300 fix(driver): update wrapErr to ignore number and boolean values (#20172) Co-authored-by: Emily Rohrbough <[email protected]> commit 1e6f51a Author: BlueWinds <[email protected]> Date: Tue Feb 15 10:18:49 2022 -0800 release 9.5.0 [skip ci] commit 507b96f Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Feb 15 06:35:08 2022 -0600 chore: Update Chrome (stable) to 98.0.4758.102 (#20192) Co-authored-by: cypress-bot[bot] <2f0651858c6e38e0+cypress-bot[bot]@users.noreply.github.com> commit e8d7120 Merge: dff45ca bb26e98 Author: BlueWinds <[email protected]> Date: Mon Feb 14 14:15:27 2022 -0800 Merge remote-tracking branch 'origin/master' into develop commit dff45ca Merge: a3f0d63 2bad703 Author: Blue F <[email protected]> Date: Mon Feb 14 12:59:52 2022 -0800 Merge pull request #20142 from cypress-io/9b967e06f5-master-into-develop chore: merge master (9b967e0) into develop commit 2bad703 Merge: 9b967e0 a3f0d63 Author: Blue F <[email protected]> Date: Mon Feb 14 12:26:14 2022 -0800 Merge branch 'develop' into 9b967e0-master-into-develop commit ce956de Author: BlueWinds <[email protected]> Date: Mon Feb 14 08:21:15 2022 -0800 Add span names, merge develop commit ac1faf4 Merge: 961e764 2c88f0c Author: BlueWinds <[email protected]> Date: Mon Feb 14 08:19:17 2022 -0800 Merge remote-tracking branch 'origin/develop' into issue-19403-perf-reporter-changes commit 961e764 Author: BlueWinds <[email protected]> Date: Thu Feb 10 14:21:44 2022 -0800 Move sending root event to own script commit bb26e98 Author: semantic-release-bot <[email protected]> Date: Thu Feb 10 15:16:39 2022 -0500 chore: release @cypress/vue-v3.1.1 [skip ci] commit 6a96ca5 Author: semantic-release-bot <[email protected]> Date: Thu Feb 10 15:16:23 2022 -0500 chore: release @cypress/react-v5.12.3 [skip ci] commit 9b967e0 Author: Lachlan Miller <[email protected]> Date: Thu Feb 10 16:26:20 2022 +1000 fix: set correct default when using react-scripts plugin (#20141) commit e709184 Merge: e0bf811 97e6c14 Author: Barthélémy Ledoux <[email protected]> Date: Wed Feb 9 20:03:05 2022 -0600 Merge pull request #20132 from cypress-io/elevatebart/trigger-vue-release commit 97e6c14 Author: ElevateBart <[email protected]> Date: Wed Feb 9 17:52:38 2022 -0600 fix: create a dummy commit to trigger release commit fa0b68a Author: BlueWinds <[email protected]> Date: Tue Feb 8 14:50:04 2022 -0800 Fix path commit f7c46fc Author: BlueWinds <[email protected]> Date: Tue Feb 8 14:35:00 2022 -0800 Refactor async data into more convenient helper commit 57decf4 Author: BlueWinds <[email protected]> Date: Tue Feb 8 13:21:57 2022 -0800 Include honeycomb key so we can send root event commit 3cf4a2b Author: BlueWinds <[email protected]> Date: Tue Feb 8 12:55:50 2022 -0800 Reduce event duplication commit 9d433c7 Author: BlueWinds <[email protected]> Date: Tue Feb 8 12:50:24 2022 -0800 Send root honeycomb event even if node_modules cache already exists commit 705262f Author: BlueWinds <[email protected]> Date: Tue Feb 8 10:48:17 2022 -0800 Fix commit b9c0d5e Author: BlueWinds <[email protected]> Date: Tue Feb 8 10:37:52 2022 -0800 fix indentation commit f694bd2 Author: BlueWinds <[email protected]> Date: Tue Feb 8 10:33:15 2022 -0800 Another attempt to wrap all events into one trace commit 5e96b5c Author: BlueWinds <[email protected]> Date: Tue Feb 8 09:52:17 2022 -0800 Attepmt to associate all circleci jobs for a given build into one trace commit 980b951 Author: BlueWinds <[email protected]> Date: Tue Feb 8 09:34:59 2022 -0800 Tweak honeycomb events further commit b8f1b4f Merge: 0d834bc 52ed6ed Author: BlueWinds <[email protected]> Date: Tue Feb 8 09:12:20 2022 -0800 Merge remote-tracking branch 'origin/develop' into issue-19403-perf-reporter-changes commit 0d834bc Author: BlueWinds <[email protected]> Date: Mon Feb 7 12:25:35 2022 -0800 Add data to all events sent, not just top level commit ac5bfd3 Author: BlueWinds <[email protected]> Date: Mon Feb 7 11:37:55 2022 -0800 Use honeycomb tracing commit 304d8bd Author: BlueWinds <[email protected]> Date: Mon Feb 7 10:22:04 2022 -0800 Make performance-reporter for system tests use get-next-version script

BlueWinds added 2 commits February 7, 2022 10:22

Make performance-reporter for system tests use get-next-version script

304d8bd

Use honeycomb tracing

ac5bfd3

BlueWinds added 3 commits February 7, 2022 12:25

Add data to all events sent, not just top level

0d834bc

Merge remote-tracking branch 'origin/develop' into issue-19403-perf-r…

b8f1b4f

…eporter-changes

Tweak honeycomb events further

980b951

BlueWinds changed the title ~~Issue 19403 perf reporter changes~~ chore: Issue 19403 perf reporter changes Feb 8, 2022

BlueWinds changed the title ~~chore: Issue 19403 perf reporter changes~~ chore: Performance reporter changes Feb 8, 2022

BlueWinds added 9 commits February 8, 2022 09:52

Attepmt to associate all circleci jobs for a given build into one trace

5e96b5c

Another attempt to wrap all events into one trace

f694bd2

fix indentation

b9c0d5e

Fix

705262f

Send root honeycomb event even if node_modules cache already exists

9d433c7

Reduce event duplication

3cf4a2b

Include honeycomb key so we can send root event

57decf4

Refactor async data into more convenient helper

f7c46fc

Fix path

fa0b68a

BlueWinds commented Feb 9, 2022

View reviewed changes

tbiethman requested review from tgriesser and flotwig February 9, 2022 16:43

BlueWinds marked this pull request as ready for review February 9, 2022 17:01

BlueWinds requested a review from a team as a code owner February 9, 2022 17:01

flotwig previously approved these changes Feb 10, 2022

View reviewed changes

Move sending root event to own script

961e764

BlueWinds dismissed flotwig’s stale review via 961e764 February 10, 2022 22:21

BlueWinds requested a review from flotwig February 10, 2022 22:22

flotwig previously approved these changes Feb 10, 2022

View reviewed changes

tgriesser requested changes Feb 11, 2022

View reviewed changes

BlueWinds added 2 commits February 14, 2022 08:19

Merge remote-tracking branch 'origin/develop' into issue-19403-perf-r…

ac1faf4

…eporter-changes

Add span names, merge develop

ce956de

BlueWinds dismissed flotwig’s stale review via ce956de February 14, 2022 16:21

BlueWinds requested a review from tgriesser February 14, 2022 20:38

flotwig self-requested a review February 16, 2022 16:37

flotwig approved these changes Feb 16, 2022

View reviewed changes

BlueWinds merged commit e55974c into develop Feb 16, 2022

BlueWinds deleted the issue-19403-perf-reporter-changes branch February 16, 2022 17:47

cypress-bot bot mentioned this pull request Feb 16, 2022

Unification: Performance Testing #19403

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: Performance reporter changes #20079

chore: Performance reporter changes #20079

Uh oh!

BlueWinds commented Feb 7, 2022 •

edited

Loading

Uh oh!

cypress-bot bot commented Feb 7, 2022

Uh oh!

cypress bot commented Feb 7, 2022 •

edited

Loading

Uh oh!

BlueWinds Feb 9, 2022

Uh oh!

flotwig left a comment

Uh oh!

flotwig Feb 10, 2022

Uh oh!

BlueWinds Feb 10, 2022

Uh oh!

tgriesser left a comment

Uh oh!

BlueWinds commented Feb 14, 2022

Uh oh!

BlueWinds commented Feb 16, 2022

Uh oh!

Uh oh!

chore: Performance reporter changes #20079

chore: Performance reporter changes #20079

Uh oh!

Conversation

BlueWinds commented Feb 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User facing changelog

Additional details

How has the user experience changed?

PR Tasks

Uh oh!

cypress-bot bot commented Feb 7, 2022

Uh oh!

cypress bot commented Feb 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test summary

Run details

Uh oh!

BlueWinds Feb 9, 2022

Choose a reason for hiding this comment

Uh oh!

flotwig left a comment

Choose a reason for hiding this comment

Uh oh!

flotwig Feb 10, 2022

Choose a reason for hiding this comment

Uh oh!

BlueWinds Feb 10, 2022

Choose a reason for hiding this comment

Uh oh!

tgriesser left a comment

Choose a reason for hiding this comment

Uh oh!

BlueWinds commented Feb 14, 2022

Uh oh!

BlueWinds commented Feb 16, 2022

Uh oh!

Uh oh!

BlueWinds commented Feb 7, 2022 •

edited

Loading

cypress bot commented Feb 7, 2022 •

edited

Loading