Avi Drissman | dfd88085 | 2022-09-15 20:11:09 | [diff] [blame] | 1 | <!-- Copyright 2020 The Chromium Authors |
Juan Antonio Navarro Perez | 897455e | 2019-02-27 00:06:03 | [diff] [blame] | 2 | Use of this source code is governed by a BSD-style license that can be |
| 3 | found in the LICENSE file. |
| 4 | --> |
| 5 | |
Caleb Rouleau | e274b5896 | 2020-03-07 01:45:17 | [diff] [blame] | 6 | # Chrome Benchmarking System |
| 7 | |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 8 | # Overview |
| 9 | |
Caleb Rouleau | e274b5896 | 2020-03-07 01:45:17 | [diff] [blame] | 10 | This directory contains benchmarks and infrastructure to test Chrome and |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 11 | Chromium and output performance measurements. These benchmarks are continuously |
Ryan Heise | f7805bb | 2022-03-16 00:18:36 | [diff] [blame] | 12 | run on the [perf waterfall](https://ci.chromium.org/p/chrome/g/chrome.perf/console). |
Caleb Rouleau | e274b5896 | 2020-03-07 01:45:17 | [diff] [blame] | 13 | |
Paul Irish | fd5159a | 2023-03-11 00:57:07 | [diff] [blame] | 14 | Also, see more information on [how Chrome measures performance](/docs/speed/how_does_chrome_measure_performance.md). |
Caleb Rouleau | e274b5896 | 2020-03-07 01:45:17 | [diff] [blame] | 15 | |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 16 | # Using The Chrome Benchmarking System |
Caleb Rouleau | e274b5896 | 2020-03-07 01:45:17 | [diff] [blame] | 17 | |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 18 | ## Analyzing Results From The Perf Waterfall |
Caleb Rouleau | e274b5896 | 2020-03-07 01:45:17 | [diff] [blame] | 19 | |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 20 | The [ChromePerf Dashboard](https://chromeperf.appspot.com/) is the destination |
| 21 | for all metrics generated by the perf waterfall. It provides tools to set up a |
| 22 | dashboard for performance of a set of tests + metrics over time. In addition, it |
| 23 | provides the ability to launch a bisection by selecting a point on the |
| 24 | dashboard. |
Caleb Rouleau | e274b5896 | 2020-03-07 01:45:17 | [diff] [blame] | 25 | |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 26 | ## Running A Single Test |
Caleb Rouleau | e274b5896 | 2020-03-07 01:45:17 | [diff] [blame] | 27 | |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 28 | The Chrome Benchmarking System has two methods for manually running performance tests: |
Ryan Heise | f7805bb | 2022-03-16 00:18:36 | [diff] [blame] | 29 | run_benchmark and Pinpoint. |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 30 | |
| 31 | run_benchmark is useful for creating and debugging benchmarks using local |
| 32 | devices. Run from the command line, it has a number of flags useful for |
| 33 | determining the internal state of the benchmark. For more information, see |
Paul Irish | fd5159a | 2023-03-11 00:57:07 | [diff] [blame] | 34 | [Run Benchmarks Locally](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry/docs/run_benchmarks_locally.md). |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 35 | |
| 36 | [Pinpoint](https://pinpoint-dot-chromeperf.appspot.com/) wraps run_benchmark and |
| 37 | provides the ability to remotely run A/B benchmarks using any platform available |
| 38 | in our lab. It will run a benchmark for as many iterations as needed to get a |
| 39 | statistically significant result, then visualize it. |
| 40 | |
Ryan Heise | 1eb5728 | 2020-10-28 17:20:17 | [diff] [blame] | 41 | If your're trying to debug a test or figure out how the infrastructure works, |
| 42 | the easiest way is to set up the debugger in VSCode (guide |
| 43 | [here](../../docs/vscode_python.md))] and set a breakpoint in |
| 44 | /tools/perf/core/benchmark_runner.py. |
| 45 | |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 46 | ## Creating New Tests (stories) |
| 47 | |
Paul Irish | fd5159a | 2023-03-11 00:57:07 | [diff] [blame] | 48 | [The telemetry readme](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry) |
Bastian Kersting | 1057d74 | 2022-11-14 09:26:57 | [diff] [blame] | 49 | provides an overview of how tests are structured and some of the underlying |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 50 | technologies. After reading that doc, figure out if your story fits into an |
| 51 | existing benchmark by checking |
| 52 | [here](https://goto.google.com/chrome-benchmarking-sheet) (or |
Ryan Heise | f7805bb | 2022-03-16 00:18:36 | [diff] [blame] | 53 | [here](https://bit.ly/chrome-benchmarks) for non-Googlers). |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 54 | |
| 55 | * If it does, follow the instructions next to it. If there are no instructions, |
| 56 | find the test type in src/tools/perf/page_sets. |
Paul Irish | fd5159a | 2023-03-11 00:57:07 | [diff] [blame] | 57 | * Otherwise, read the [Guideline for adding new benchmark](https://docs.google.com/document/d/1ni2MIeVnlH4bTj4yvEDMVNxgL73PqK_O9_NUm3NW3BA/edit). |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 58 | |
| 59 | After figuring out where your story fits, create a new one. There is a |
| 60 | considerable amount of variation between different benchmarks, so use a nearby |
| 61 | story as a model. You may also need to introduce custom JavaScript to drive |
| 62 | interactions on the page or to deal with nondeterminsim. For an example, search |
John Palmer | ab8812a | 2021-05-21 17:03:43 | [diff] [blame] | 63 | [this file](https://source.chromium.org/chromium/chromium/src/+/main:tools/perf/page_sets/system_health/browsing_stories.py?q=browsing_stories.py&ss=chromium) |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 64 | for browse:tools:sheets:2019. |
| 65 | |
Ryan Heise | f7805bb | 2022-03-16 00:18:36 | [diff] [blame] | 66 | Next, we need to create a recording of all the content requested by the test. We then use this by serving it from a web server running on either the test device or a host device (for Android tests), removing one more source of nondeterminism. To do so, follow [these instructions](https://source.chromium.org/chromium/chromium/src/+/main:tools/perf/recording_benchmarks.md). |
Ryan Heise | 4c3160e | 2020-10-06 00:48:52 | [diff] [blame] | 67 | |
| 68 | # Tools In This Directory |
Juan Antonio Navarro Perez | 897455e | 2019-02-27 00:06:03 | [diff] [blame] | 69 | |
Dean Michael Berris | 6517fa74 | 2020-04-27 02:59:15 | [diff] [blame] | 70 | This directory contains a variety of tools that can be used to run benchmarks, |
| 71 | interact with speed services, and manage performance waterfall configurations. |
| 72 | It also has commands for running functional unittests. |
Juan Antonio Navarro Perez | 897455e | 2019-02-27 00:06:03 | [diff] [blame] | 73 | |
Paul Irish | fd5159a | 2023-03-11 00:57:07 | [diff] [blame] | 74 | ## `run_tests` |
Caleb Rouleau | 83c95a5 | 2019-11-23 02:32:30 | [diff] [blame] | 75 | |
| 76 | This command allows you to run functional tests against the python code in this |
| 77 | directory. For example, try: |
| 78 | |
| 79 | ``` |
| 80 | ./run_tests results_dashboard_unittest |
| 81 | ``` |
| 82 | |
| 83 | Note that the positional argument can be any substring within the test name. |
| 84 | |
| 85 | This may require you to set up your `gsutil config` first. |
| 86 | |
Paul Irish | fd5159a | 2023-03-11 00:57:07 | [diff] [blame] | 87 | ## `run_benchmark` |
Juan Antonio Navarro Perez | 897455e | 2019-02-27 00:06:03 | [diff] [blame] | 88 | |
| 89 | This command allows running benchmarks defined in the chromium repository, |
| 90 | specifically in [tools/perf/benchmarks][benchmarks_dir]. If you need it, |
Dean Michael Berris | 6517fa74 | 2020-04-27 02:59:15 | [diff] [blame] | 91 | documentation is available on how to [run benchmarks locally][run_locally] and |
| 92 | how to properly [set up your device][device_setup]. |
Juan Antonio Navarro Perez | 897455e | 2019-02-27 00:06:03 | [diff] [blame] | 93 | |
| 94 | [benchmarks_dir]: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/ |
| 95 | [run_locally]: https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry/docs/run_benchmarks_locally.md |
Paul Irish | fd5159a | 2023-03-11 00:57:07 | [diff] [blame] | 96 | [device_setup]: https://source.chromium.org/chromium/chromium/src/+/main:docs/speed/benchmark/telemetry_device_setup.md |
Juan Antonio Navarro Perez | 897455e | 2019-02-27 00:06:03 | [diff] [blame] | 97 | |
Paul Irish | fd5159a | 2023-03-11 00:57:07 | [diff] [blame] | 98 | ## `update_wpr` |
Juan Antonio Navarro Perez | 897455e | 2019-02-27 00:06:03 | [diff] [blame] | 99 | |
Juan Antonio Navarro Perez | 2788bfb | 2019-08-08 20:53:46 | [diff] [blame] | 100 | A helper script to automate various tasks related to the update of |
| 101 | [Web Page Recordings][wpr] for our benchmarks. In can help creating new |
Dean Michael Berris | 6517fa74 | 2020-04-27 02:59:15 | [diff] [blame] | 102 | recordings from live websites, replay those to make sure they work, upload them |
| 103 | to cloud storage, and finally send a CL to review with the new recordings. |
Juan Antonio Navarro Perez | 897455e | 2019-02-27 00:06:03 | [diff] [blame] | 104 | |
Camillo Bruni | 617bba1 | 2024-02-27 21:29:21 | [diff] [blame] | 105 | [wpr]: https://github.com/catapult-project/catapult/tree/master/web_page_replay_go |