Merge GPU generate_buildbot_json.py into testing/buildbot version.
Redefine the chromium.gpu, chromium.gpu.fyi and client.v8.fyi
waterfalls and test suites in waterfalls.pyl and test_suites.pyl,
respectively.
The goal of this conversion was to generate these three waterfalls'
JSON files almost exactly as they currently are, which has been
achieved. There are however many cleanups that should be done to
simplify the test suites and the configurations of the GPU bots, which
will be done in follow-on CLs and bugs.
Added a few needed features to the testing/buildbot version of the
generator script. Covered these in the unit tests; the coverage test
continues to pass. Deleted the old GPU script and updated docs that
referred to it.
Tbr: [email protected]
Tbr: [email protected]
Tbr: [email protected]
Tbr: [email protected]
Tbr: [email protected]
Bug: 844815
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I4e472d0e55a700fe3cec5043dcd260d31de8fd41
Reviewed-on: https://chromium-review.googlesource.com/1067132
Reviewed-by: Kenneth Russell <[email protected]>
Commit-Queue: Kenneth Russell <[email protected]>
Cr-Commit-Position: refs/heads/master@{#563939}diff --git a/docs/gpu/gpu_testing.md b/docs/gpu/gpu_testing.md
index 5b05f8a..b4a925e 100644
--- a/docs/gpu/gpu_testing.md
+++ b/docs/gpu/gpu_testing.md
@@ -76,8 +76,8 @@
<!-- XXX: broken link -->
[new-testing-infra]: https://github.com/luci/luci-py/wiki
[isolated-testing-infra]: https://www.chromium.org/developers/testing/isolated-testing/infrastructure
-[chromium.gpu]: https://build.chromium.org/p/chromium.gpu/console
-[chromium.gpu.fyi]: https://build.chromium.org/p/chromium.gpu.fyi/console
+[chromium.gpu]: https://ci.chromium.org/p/chromium/g/chromium.gpu/console
+[chromium.gpu.fyi]: https://ci.chromium.org/p/chromium/g/chromium.gpu.fyi/console
[tools/build workspace]: https://code.google.com/p/chromium/codesearch#chromium/build/scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py
[bots-presentation]: https://docs.google.com/presentation/d/1BC6T7pndSqPFnituR7ceG7fMY7WaGqYHhx5i9ECa8EI/edit?usp=sharing
@@ -408,27 +408,33 @@
These files are autogenerated by the following script:
-* [`generate_buildbot_json.py`](https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/generate_buildbot_json.py)
+* [`generate_buildbot_json.py`](https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/generate_buildbot_json.py)
-This script is completely self-contained and should hopefully be
-self-explanatory. The JSON files are parsed by the chromium and chromium_trybot
-recipes, and describe two types of tests:
+This script is documented in
+[`testing/buildbot/README.md`](https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/README.md). The
+JSON files are parsed by the chromium and chromium_trybot recipes, and describe
+two basic types of tests:
* GTests: those which use the Googletest and Chromium's `base/test/launcher/`
frameworks.
-* Telemetry based tests: those which are built on the Telemetry framework and
- launch the entire browser.
+* Isolated scripts: tests whose initial entry point is a Python script which
+ follows a simple convention of command line argument parsing.
+
+The majority of the GPU tests are however:
+
+* Telemetry based tests: an isolated script test which is built on the
+ Telemetry framework and which launches the entire browser.
A prerequisite of adding a new test to the bots is that that test [run via
-isolates][new-isolates]. Once that is done, modify `generate_buildbot_json.py` to add the
-test to the appropriate set of bots. Be careful when adding large new test
-steps to all of the bots, because the GPU bots are a limited resource and do
-not currently have the capacity to absorb large new test suites. It is safer to
-get new tests running on the chromium.gpu.fyi waterfall first, and expand from
-there to the chromium.gpu waterfall (which will also make them run against
-every Chromium CL by virtue of the `linux_chromium_rel_ng`,
-`mac_chromium_rel_ng` and `win_chromium_rel_ng` tryservers' mirroring of the
-bots on this waterfall – so be careful!).
+isolates][new-isolates]. Once that is done, modify `test_suites.pyl` to add the
+test to the appropriate set of bots. Be careful when adding large new test steps
+to all of the bots, because the GPU bots are a limited resource and do not
+currently have the capacity to absorb large new test suites. It is safer to get
+new tests running on the chromium.gpu.fyi waterfall first, and expand from there
+to the chromium.gpu waterfall (which will also make them run against every
+Chromium CL by virtue of the `linux_chromium_rel_ng`, `mac_chromium_rel_ng`,
+`win7_chromium_rel_ng` and `android-marshmallow-arm64-rel` tryservers' mirroring
+of the bots on this waterfall – so be careful!).
Tryjobs which add new test steps to the chromium.gpu.json file will run those
new steps during the tryjob, which helps ensure that the new test won't break
