Add "exclusive_tests" field in VirtualTestSuites
Previously for a virtual test to skip the base tests, we needed to add
the following entries in NeverFixTests:
base/tests/* [ Skip ]
virtual/suite/base/tests/* [ Pass ]
and we needed to keep them in sync with VirtualTestSuites.
With the "exclusive_tests" field, we can define a virtual suite and
skip all or selected base tests at one place, which is easier to keep
consistency.
Bug: 1379095
Change-Id: Ic76a76eadcca3b7786906a925e5011c43ed59ce0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4010728
Reviewed-by: Weizhong Xia <[email protected]>
Commit-Queue: Xianzhu Wang <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1069366}
diff --git a/docs/testing/web_tests.md b/docs/testing/web_tests.md
index ce40024..a11fd45 100644
--- a/docs/testing/web_tests.md
+++ b/docs/testing/web_tests.md
@@ -313,6 +313,39 @@
in case that we just want to run the real tests under `virtual/<prefix>`
with the flags without creating any virtual tests.
+A virtual test suite can have an optional `exclusive_tests` field to specify
+all (with `"ALL"`) or a subset of `bases` tests that will be exclusively run
+under this virtual suite. The specified base tests will be skipped. Corresponding
+virtual tests under other virtual suites that don't specify the tests in their
+`exclusive_tests` list will be skipped, too. For example (unrelated fields
+are omitted):
+
+```json
+{
+ "prefix": "v1",
+ "bases": ["a"],
+}
+{
+ "prefix": "v2",
+ "bases": ["a/a1", "a/a2"],
+ "exclusive_tests": "ALL",
+}
+{
+ "prefix": "v3",
+ "bases": ["a"],
+ "exclusive_tests": ["a/a1"],
+}
+```
+Suppose there are directories `a/a1`, `a/a2` and `a/a3`, we will run the
+following tests:
+| Suite | a/a1 | a/a2 | a/a3 |
+| ---------: | :-----: | :-----: | :--: |
+| base | skipped | skipped | run |
+| virtual/v1 | skipped | skipped | run |
+| virtual/v2 | run | run | n/a |
+| virtual/v3 | run | skipped | run |
+
+
### Choosing between flag-specific and virtual test suite
For flags whose implementation is still in progress, flag-specific expectations
