web_tests/FlagSpecificConfig
It specifies the name of the flag-specific expectation file
under web_tests/FlagExpectations and the baseline directory under
web_tests/flag-specific, in the following format:
{
"name": "short-name",
"args": ["--arg1", "--arg2"]
}
When at least --additional-driver-flag=--arg1 and
--additional-driver-flag=--arg2 are in run_web_tests.py command line,
or --flag-specific=short-name is in the command line,
we will find web_tests/FlagExpectations/short-name for the additional
expectation file and web_tests/flag-specific/short-name for the
additional baseline directory.
Bug: 1019501
Change-Id: Idf0621abc89efc18cd17f9a9612074aa9d7de298
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1894069
Commit-Queue: Xianzhu Wang <[email protected]>
Reviewed-by: Steve Kobes <[email protected]>
Cr-Commit-Position: refs/heads/master@{#712013}
diff --git a/docs/testing/web_tests.md b/docs/testing/web_tests.md
index 6ea10f0b..3ab8de8 100644
--- a/docs/testing/web_tests.md
+++ b/docs/testing/web_tests.md
@@ -68,11 +68,12 @@
Tests marked as `[ Skip ]` in
[TestExpectations](../../third_party/blink/web_tests/TestExpectations)
-won't be run at all, generally because they cause some intractable tool error.
+won't be run by default, generally because they cause some intractable tool error.
To force one of them to be run, either rename that file or specify the skipped
-test as the only one on the command line (see below). Read the
-[Web Test Expectations documentation](./web_test_expectations.md) to learn
-more about TestExpectations and related files.
+test on the command line (see below) or in a file specified with --test-list
+(however, --skip=always can make the tests marked as `[ Skip ]` always skipped).
+Read the [Web Test Expectations documentation](./web_test_expectations.md) to
+learn more about TestExpectations and related files.
*** promo
Currently only the tests listed in
@@ -220,6 +221,31 @@
`web_tests/FlagExpectations/blocking-repaint`, if this file exists. The
suppressions in this file override the main TestExpectations file.
+ It will also look for baselines in `web_tests/flag-specific/blocking-repaint`.
+ The baselines in this directory override the fallback baselines.
+
+ By default, name of the expectation file name under
+ `web_tests/FlagExpectations` and name of the baseline directory under
+ `web_tests/flag-specific` uses the first flag of --additional-driver-flag
+ with leading '-'s stripped.
+
+ You can also customize the name in `web_tests/FlagSpecificConfig` when
+ the name is too long or when we need to match multiple additional args:
+
+ ```json
+ {
+ "name": "short-name",
+ "args": ["--blocking-repaint", "--another-flag"]
+ }
+ ```
+
+ When at least `--additional-driver-flag=--blocking-repaint` and
+ `--additional-driver-flag=--another-flag` are specified, `short-name` will
+ be used as name of the flag specific expectation file and the baseline directory.
+
+ With the config, you can also use `--flag-specific=short-name` as a shortcut
+ of `--additional-driver-flag=--blocking-repaint --additional-driver-flag=--another-flag`.
+
* Using a *virtual test suite* defined in
[web_tests/VirtualTestSuites](../../third_party/blink/web_tests/VirtualTestSuites).
A virtual test suite runs a subset of web tests with additional flags, with
