Non-headless Fuzzy Blink Web Tests
This is a split of a previous review: https://chromium-review.googlesource.com/c/chromium/src/+/1500784
There we discussed both the non-headless content_shell, as well as fuzzy pixel
diffs. However that review also had testing infrastructure changes which we are
blocked on landing.
This change pulls out the portion required for running SkiaRenderer and Vulkan
blink_web_tests. So that the team can run them locally.
content_shell defaults to a headless state when running the blink_web_tests.
Which forces Chrome to perform software rendering. There is currently no Swift
Shader equivalent for Vulkan. So we cannot use this for testing.
This change introduces a new Chrome switch "--disable-headless-mode" which
prevents the content_shell from triggering the headless state.
Actual GPU output introduces a variance from the baseline expectation files. So
this change also updates image_diff to have a new flag "--fuzzy-diff" which
allows it to have a tolerance in pixel differences. A corresponding
"--fuzzy-diff" flag has been added for blink_web_tests to be able to
trigger this.
I've also updated documentation of blink_web_tests based on findings
while writting this.
A follow up change will land tests on the Chromium GPU.FYI bots.
Bug: 925161
Change-Id: Ifd3837ec48a32bbdc776433e8c543138dde7418d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1553280
Commit-Queue: Jonathan Ross <[email protected]>
Reviewed-by: Kenneth Russell <[email protected]>
Reviewed-by: Antoine Labour <[email protected]>
Reviewed-by: Dirk Pranke <[email protected]>
Cr-Commit-Position: refs/heads/master@{#648767}diff --git a/docs/testing/web_tests.md b/docs/testing/web_tests.md
index 850a26c2..eea7c20 100644
--- a/docs/testing/web_tests.md
+++ b/docs/testing/web_tests.md
@@ -328,12 +328,12 @@
spacing or box sizes are often unimportant, especially around fonts and
form controls. Differences in wording of JS error messages are also
usually acceptable.
- * `python run_web_tests.py path/to/your/test.html --full-results-html`
- produces a page including links to the expected result, actual result,
- and diff.
- * Add the `--sources` option to `run_web_tests.py` to see exactly which
- expected result it's comparing to (a file next to the test, something in
- platform/mac/, something in platform/chromium-win/, etc.)
+ * `python run_web_tests.py path/to/your/test.html` produces a page listing
+ all test results. Those which fail their expectations will include links
+ to the expected result, actual result, and diff. These results are saved
+ to `$root_build_dir/layout-test-results`.
+ * Alternatively the `--results-directory=path/for/output/` option allows
+ you to specify an alternative directory for the output to be saved to.
* If you're still sure it's correct, rebaseline the test (see below).
Otherwise...
* If you're lucky, your test is one that runs properly when you navigate to it
