blob: a119fb7ec58ce0a4379b0bc0af3c58b88488524e [file] [log] [blame] [view]
Avi Drissmandfd880852022-09-15 20:11:091<!-- Copyright 2020 The Chromium Authors
Juan Antonio Navarro Perez897455e2019-02-27 00:06:032 Use of this source code is governed by a BSD-style license that can be
3 found in the LICENSE file.
4-->
5
Caleb Rouleaue274b58962020-03-07 01:45:176# Chrome Benchmarking System
7
Ryan Heise4c3160e2020-10-06 00:48:528# Overview
9
Caleb Rouleaue274b58962020-03-07 01:45:1710This directory contains benchmarks and infrastructure to test Chrome and
Ryan Heise4c3160e2020-10-06 00:48:5211Chromium and output performance measurements. These benchmarks are continuously
Ryan Heisef7805bb2022-03-16 00:18:3612run on the [perf waterfall](https://ci.chromium.org/p/chrome/g/chrome.perf/console).
Caleb Rouleaue274b58962020-03-07 01:45:1713
Paul Irishfd5159a2023-03-11 00:57:0714Also, see more information on [how Chrome measures performance](/docs/speed/how_does_chrome_measure_performance.md).
Caleb Rouleaue274b58962020-03-07 01:45:1715
Ryan Heise4c3160e2020-10-06 00:48:5216# Using The Chrome Benchmarking System
Caleb Rouleaue274b58962020-03-07 01:45:1717
Ryan Heise4c3160e2020-10-06 00:48:5218## Analyzing Results From The Perf Waterfall
Caleb Rouleaue274b58962020-03-07 01:45:1719
Ryan Heise4c3160e2020-10-06 00:48:5220The [ChromePerf Dashboard](https://chromeperf.appspot.com/) is the destination
21for all metrics generated by the perf waterfall. It provides tools to set up a
22dashboard for performance of a set of tests + metrics over time. In addition, it
23provides the ability to launch a bisection by selecting a point on the
24dashboard.
Caleb Rouleaue274b58962020-03-07 01:45:1725
Ryan Heise4c3160e2020-10-06 00:48:5226## Running A Single Test
Caleb Rouleaue274b58962020-03-07 01:45:1727
Ryan Heise4c3160e2020-10-06 00:48:5228The Chrome Benchmarking System has two methods for manually running performance tests:
Ryan Heisef7805bb2022-03-16 00:18:3629run_benchmark and Pinpoint.
Ryan Heise4c3160e2020-10-06 00:48:5230
31run_benchmark is useful for creating and debugging benchmarks using local
32devices. Run from the command line, it has a number of flags useful for
33determining the internal state of the benchmark. For more information, see
Paul Irishfd5159a2023-03-11 00:57:0734[Run Benchmarks Locally](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry/docs/run_benchmarks_locally.md).
Ryan Heise4c3160e2020-10-06 00:48:5235
36[Pinpoint](https://pinpoint-dot-chromeperf.appspot.com/) wraps run_benchmark and
37provides the ability to remotely run A/B benchmarks using any platform available
38in our lab. It will run a benchmark for as many iterations as needed to get a
39statistically significant result, then visualize it.
40
Ryan Heise1eb57282020-10-28 17:20:1741If your're trying to debug a test or figure out how the infrastructure works,
42the easiest way is to set up the debugger in VSCode (guide
43[here](../../docs/vscode_python.md))] and set a breakpoint in
44/tools/perf/core/benchmark_runner.py.
45
Ryan Heise4c3160e2020-10-06 00:48:5246## Creating New Tests (stories)
47
Paul Irishfd5159a2023-03-11 00:57:0748[The telemetry readme](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry)
Bastian Kersting1057d742022-11-14 09:26:5749provides an overview of how tests are structured and some of the underlying
Ryan Heise4c3160e2020-10-06 00:48:5250technologies. After reading that doc, figure out if your story fits into an
51existing benchmark by checking
52[here](https://goto.google.com/chrome-benchmarking-sheet) (or
Ryan Heisef7805bb2022-03-16 00:18:3653[here](https://bit.ly/chrome-benchmarks) for non-Googlers).
Ryan Heise4c3160e2020-10-06 00:48:5254
55* If it does, follow the instructions next to it. If there are no instructions,
56 find the test type in src/tools/perf/page_sets.
Paul Irishfd5159a2023-03-11 00:57:0757* Otherwise, read the [Guideline for adding new benchmark](https://docs.google.com/document/d/1ni2MIeVnlH4bTj4yvEDMVNxgL73PqK_O9_NUm3NW3BA/edit).
Ryan Heise4c3160e2020-10-06 00:48:5258
59After figuring out where your story fits, create a new one. There is a
60considerable amount of variation between different benchmarks, so use a nearby
61story as a model. You may also need to introduce custom JavaScript to drive
62interactions on the page or to deal with nondeterminsim. For an example, search
John Palmerab8812a2021-05-21 17:03:4363[this file](https://source.chromium.org/chromium/chromium/src/+/main:tools/perf/page_sets/system_health/browsing_stories.py?q=browsing_stories.py&ss=chromium)
Ryan Heise4c3160e2020-10-06 00:48:5264for browse:tools:sheets:2019.
65
Ryan Heisef7805bb2022-03-16 00:18:3666Next, we need to create a recording of all the content requested by the test. We then use this by serving it from a web server running on either the test device or a host device (for Android tests), removing one more source of nondeterminism. To do so, follow [these instructions](https://source.chromium.org/chromium/chromium/src/+/main:tools/perf/recording_benchmarks.md).
Ryan Heise4c3160e2020-10-06 00:48:5267
68# Tools In This Directory
Juan Antonio Navarro Perez897455e2019-02-27 00:06:0369
Dean Michael Berris6517fa742020-04-27 02:59:1570This directory contains a variety of tools that can be used to run benchmarks,
71interact with speed services, and manage performance waterfall configurations.
72It also has commands for running functional unittests.
Juan Antonio Navarro Perez897455e2019-02-27 00:06:0373
Paul Irishfd5159a2023-03-11 00:57:0774## `run_tests`
Caleb Rouleau83c95a52019-11-23 02:32:3075
76This command allows you to run functional tests against the python code in this
77directory. For example, try:
78
79```
80./run_tests results_dashboard_unittest
81```
82
83Note that the positional argument can be any substring within the test name.
84
85This may require you to set up your `gsutil config` first.
86
Paul Irishfd5159a2023-03-11 00:57:0787## `run_benchmark`
Juan Antonio Navarro Perez897455e2019-02-27 00:06:0388
89This command allows running benchmarks defined in the chromium repository,
90specifically in [tools/perf/benchmarks][benchmarks_dir]. If you need it,
Dean Michael Berris6517fa742020-04-27 02:59:1591documentation is available on how to [run benchmarks locally][run_locally] and
92how to properly [set up your device][device_setup].
Juan Antonio Navarro Perez897455e2019-02-27 00:06:0393
94[benchmarks_dir]: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/
95[run_locally]: https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry/docs/run_benchmarks_locally.md
Paul Irishfd5159a2023-03-11 00:57:0796[device_setup]: https://source.chromium.org/chromium/chromium/src/+/main:docs/speed/benchmark/telemetry_device_setup.md
Juan Antonio Navarro Perez897455e2019-02-27 00:06:0397
Paul Irishfd5159a2023-03-11 00:57:0798## `update_wpr`
Juan Antonio Navarro Perez897455e2019-02-27 00:06:0399
Juan Antonio Navarro Perez2788bfb2019-08-08 20:53:46100A helper script to automate various tasks related to the update of
101[Web Page Recordings][wpr] for our benchmarks. In can help creating new
Dean Michael Berris6517fa742020-04-27 02:59:15102recordings from live websites, replay those to make sure they work, upload them
103to cloud storage, and finally send a CL to review with the new recordings.
Juan Antonio Navarro Perez897455e2019-02-27 00:06:03104
Camillo Bruni617bba12024-02-27 21:29:21105[wpr]: https://github.com/catapult-project/catapult/tree/master/web_page_replay_go