pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 1 | # Writing Layout Tests |
| 2 | |
| 3 | _Layout tests_ is a bit of a misnomer. This term is |
| 4 | [a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory/), |
| 5 | and we use it to refer to every test that is written as a Web page (HTML, SVG, |
| 6 | or XHTML) and lives in |
| 7 | [third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests). |
| 8 | |
| 9 | [TOC] |
| 10 | |
| 11 | ## Overview |
| 12 | |
| 13 | Layout tests should be used to accomplish one of the following goals: |
| 14 | |
| 15 | 1. The entire surface of Blink that is exposed to the Web should be covered by |
| 16 | tests that we contribute to the |
| 17 | [Web Platform Tests Project](https://github.com/w3c/web-platform-tests) |
| 18 | (WPT). This helps us avoid regressions, and helps us identify Web Platform |
| 19 | areas where the major browsers don't have interoperable implementations. |
| 20 | Furthermore, by contributing to projects such as WPT, we share the burden of |
| 21 | writing tests with the other browser vendors, and we help all the browsers |
| 22 | get better. This is very much in line with our goal to move the Web forward. |
| 23 | 2. When a Blink feature cannot be tested using the tools provided by WPT, and |
| 24 | cannot be easily covered by |
| 25 | [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Source/web/tests/?q=webframetest&sq=package:chromium&type=cs), |
| 26 | the feature must be covered by layout tests, to avoid unexpected regressions. |
| 27 | These tests will use Blink-specific testing APIs that are only available in |
| 28 | [content_shell](./layout_tests_in_content_shell.md). |
| 29 | |
| 30 | *** promo |
| 31 | If you know that Blink layout tests are upstreamed to other projects, such as |
| 32 | [test262](https://github.com/tc39/test262), please update this document. Most |
| 33 | importantly, our guidelines should to make it easy for our tests to be |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 34 | upstreamed. The |
| 35 | [blink-dev mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/blink-dev) |
| 36 | will be happy to help you harmonize our current guidelines with communal test |
| 37 | repositories. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 38 | *** |
| 39 | |
| 40 | ### Test Types |
| 41 | |
| 42 | There are four broad types of layout tests, listed in the order of preference. |
| 43 | |
| 44 | * *JavaScript Tests* are the layout test implementation of |
| 45 | [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain |
| 46 | assertions written in JavaScript, and pass if the assertions evaluate to |
| 47 | true. |
| 48 | * *Reference Tests* render a test page and a reference page, and pass if the two |
| 49 | renderings are identical, according to a pixel-by-pixel comparison. These |
| 50 | tests are less robust, harder to debug, and significantly slower than |
| 51 | JavaScript tests, and are only used when JavaScript tests are insufficient, |
| 52 | such as when testing paint code. |
| 53 | * *Pixel Tests* render a test page and compare the result against a pre-rendered |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 54 | baseline image in the repository. Pixel tests are less robust than the |
| 55 | first two types, because the rendering of a page is influenced by |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 56 | many factors such as the host computer's graphics card and driver, the |
| 57 | platform's text rendering system, and various user-configurable operating |
| 58 | system settings. For this reason, it is common for a pixel test to have a |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 59 | different reference image for each platform that Blink is tested on, and |
| 60 | the reference images are |
| 61 | [quite cumbersome to manage](./layout_test_expectations.md). You |
| 62 | should only write a pixel test if you cannot use a reference test. By default |
| 63 | a pixel test will also dump the layout tree as text output, so they are |
| 64 | similar to ... |
| 65 | * *Layout tree tests*, which output a textual representation of the layout |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 66 | tree, which is the key data structure in Blink's page rendering system. The |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 67 | test passes if the output matches a baseline text file in the repository. |
| 68 | Layout tree tests are used as a last resort to test the internal quirks of |
| 69 | the implementation, and they should be avoided in favor of one of the earlier |
| 70 | options. |
pwnall | 59aadcb | 2017-01-26 23:27:21 | [diff] [blame^] | 71 | |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 72 | ## General Principles |
| 73 | |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 74 | |
| 75 | Tests should be written under the assumption that they will be upstreamed |
pwnall | 59aadcb | 2017-01-26 23:27:21 | [diff] [blame^] | 76 | to the WPT project. To this end, tests should follow the |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 77 | [WPT guidelines](http://testthewebforward.org/docs/writing-tests.html). |
| 78 | |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 79 | |
pwnall | 59aadcb | 2017-01-26 23:27:21 | [diff] [blame^] | 80 | There is no style guide that applies to all layout tests. However, some projects |
| 81 | have adopted style guides, such as the |
| 82 | [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing). |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 83 | |
pwnall | 59aadcb | 2017-01-26 23:27:21 | [diff] [blame^] | 84 | Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most |
| 85 | important WPT guidelines and highlights some JavaScript concepts that are worth |
| 86 | paying attention to when trying to infer style rules from existing tests. If |
| 87 | you're unopinionated and looking for a style guide to follow, the document also |
| 88 | suggests some defaults. |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 89 | |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 90 | ## JavaScript Tests |
| 91 | |
| 92 | Whenever possible, the testing criteria should be expressed in JavaScript. The |
| 93 | alternatives, which will be described in future sections, result in slower and |
| 94 | less reliable tests. |
| 95 | |
| 96 | All new JavaScript tests should be written using the |
| 97 | [testharness.js](https://github.com/w3c/testharness.js/) testing framework. This |
| 98 | framework is used by the tests in the |
| 99 | [web-platform-tests](https://github.com/w3c/web-platform-tests) repository, |
| 100 | which is shared with all the other browser vendors, so `testharness.js` tests |
| 101 | are more accessible to browser developers. |
| 102 | |
| 103 | As a shared framework, `testharness.js` enjoys high-quality documentation, such |
| 104 | as [a tutorial](http://testthewebforward.org/docs/testharness-tutorial.html) and |
| 105 | [API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.md). |
| 106 | Layout tests should follow the recommendations of the above documents. |
| 107 | Furthermore, layout tests should include relevant |
| 108 | [metadata](http://testthewebforward.org/docs/css-metadata.html). The |
| 109 | specification URL (in `<link rel="help">`) is almost always relevant, and is |
| 110 | incredibly helpful to a developer who needs to understand the test quickly. |
| 111 | |
| 112 | Below is a skeleton for a JavaScript test embedded in an HTML page. Note that, |
| 113 | in order to follow the minimality guideline, the test omits the tags `<html>`, |
| 114 | `<head>`, and `<body>`, as they can be inferred by the HTML parser. |
| 115 | |
| 116 | ```html |
| 117 | <!doctype html> |
pwnall | 59aadcb | 2017-01-26 23:27:21 | [diff] [blame^] | 118 | <title>JavaScript: the true literal is immutable and equal to itself</title> |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 119 | <link rel="help" href="https://tc39.github.io/ecma262/#sec-boolean-literals"> |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 120 | <script src="/resources/testharness.js"></script> |
| 121 | <script src="/resources/testharnessreport.js"></script> |
| 122 | <script> |
| 123 | 'use strict'; |
| 124 | |
| 125 | // Synchronous test example. |
| 126 | test(() => { |
| 127 | const value = true; |
| 128 | assert_true(value, 'true literal'); |
| 129 | assert_equals(value.toString(), 'true', 'the string representation of true'); |
| 130 | }, 'The literal true in a synchronous test case'); |
| 131 | |
| 132 | // Asynchronous test example. |
| 133 | async_test(t => { |
| 134 | const originallyTrue = true; |
| 135 | setTimeout(t.step_func_done(() => { |
| 136 | assert_equals(originallyTrue, true); |
| 137 | }), 0); |
| 138 | }, 'The literal true in a setTimeout callback'); |
| 139 | |
| 140 | // Promise test example. |
| 141 | promise_test(() => { |
| 142 | return new Promise((resolve, reject) => { |
| 143 | resolve(true); |
| 144 | }).then(value => { |
| 145 | assert_true(value); |
| 146 | }); |
| 147 | }, 'The literal true used to resolve a Promise'); |
| 148 | |
| 149 | </script> |
| 150 | ``` |
| 151 | |
| 152 | Some points that are not immediately obvious from the example: |
| 153 | |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 154 | * When calling an `assert_` function that compares two values, the first |
| 155 | argument is the actual value (produced by the functionality being tested), and |
| 156 | the second argument is the expected value (known good, golden). The order |
| 157 | is important, because the testing harness relies on it to generate expressive |
| 158 | error messages that are relied upon when debugging test failures. |
| 159 | * The assertion description (the string argument to `assert_` methods) conveys |
| 160 | the way the actual value was obtained. |
| 161 | * If the expected value doesn't make it clear, the assertion description |
| 162 | should explain the desired behavior. |
| 163 | * Test cases with a single assertion should omit the assertion's description |
| 164 | when it is sufficiently clear. |
| 165 | * Each test case describes the circumstance that it tests, without being |
| 166 | redundant. |
| 167 | * Do not start test case descriptions with redundant terms like "Testing" |
| 168 | or "Test for". |
ktyliu | e0bb988 | 2017-01-10 01:47:50 | [diff] [blame] | 169 | * Test files with a single test case should omit the test case description. |
| 170 | The file's `<title>` should be sufficient to describe the scenario being |
| 171 | tested. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 172 | * Asynchronous tests have a few subtleties. |
| 173 | * The `async_test` wrapper calls its function with a test case argument that |
| 174 | is used to signal when the test case is done, and to connect assertion |
| 175 | failures to the correct test. |
| 176 | * `t.done()` must be called after all the test case's assertions have |
| 177 | executed. |
| 178 | * Test case assertions (actually, any callback code that can throw |
| 179 | exceptions) must be wrapped in `t.step_func()` calls, so that |
| 180 | assertion failures and exceptions can be traced back to the correct test |
| 181 | case. |
| 182 | * `t.step_func_done()` is a shortcut that combines `t.step_func()` with a |
| 183 | `t.done()` call. |
| 184 | |
| 185 | *** promo |
| 186 | Layout tests that load from `file://` origins must currently use relative paths |
| 187 | to point to |
| 188 | [/resources/testharness.js](../../third_party/WebKit/LayoutTests/resources/testharness.js) |
| 189 | and |
| 190 | [/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js). |
| 191 | This is contrary to the WPT guidelines, which call for absolute paths. |
| 192 | This limitation does not apply to the tests in `LayoutTests/http`, which rely on |
| 193 | an HTTP server, or to the tests in `LayoutTests/imported/wpt`, which are |
| 194 | imported from the [WPT repository](https://github.com/w3c/web-platform-tests). |
| 195 | *** |
| 196 | |
| 197 | ### WPT Supplemental Testing APIs |
| 198 | |
| 199 | Some tests simply cannot be expressed using the Web Platform APIs. For example, |
| 200 | some tests that require a user to perform a gesture, such as a mouse click, |
| 201 | cannot be implemented using Web APIs. The WPT project covers some of these cases |
| 202 | via supplemental testing APIs. |
| 203 | |
pwnall | 59aadcb | 2017-01-26 23:27:21 | [diff] [blame^] | 204 | When writing tests that rely on supplemental testing APIs, please consider the |
| 205 | cost and benefits of having the tests |
| 206 | [gracefully degrade to manual tests](./layout_tests_with_manual_fallback.md) in |
| 207 | the absence of the testing APIs. |
| 208 | |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 209 | *** promo |
| 210 | In many cases, the user gesture is not actually necessary. For example, many |
| 211 | event handling tests can use |
| 212 | [synthetic events](https://developer.mozilla.org/docs/Web/Guide/Events/Creating_and_triggering_events). |
| 213 | *** |
| 214 | |
| 215 | *** note |
| 216 | TODO: document wpt_automation. Manual tests might end up moving here. |
| 217 | *** |
| 218 | |
| 219 | ### Relying on Blink-Specific Testing APIs |
| 220 | |
| 221 | Tests that cannot be expressed using the Web Platform APIs or WPT's testing APIs |
| 222 | use Blink-specific testing APIs. These APIs are only available in |
| 223 | [content_shell](./layout_tests_in_content_shell.md), and should only be used as |
| 224 | a last resort. |
| 225 | |
| 226 | A downside of Blink-specific APIs is that they are not as well documented as the |
| 227 | Web Platform features. Learning to use a Blink-specific feature requires finding |
| 228 | other tests that use it, or reading its source code. |
| 229 | |
| 230 | For example, the most popular Blink-specific API is `testRunner`, which is |
| 231 | implemented in |
| 232 | [components/test_runner/test_runner.h](../../components/test_runner/test_runner.h) |
| 233 | and |
| 234 | [components/test_runner/test_runner.cpp](../../components/test_runner/test_runner.cpp). |
| 235 | By skimming the `TestRunnerBindings::Install` method, we learn that the |
| 236 | testRunner API is presented by the `window.testRunner` and |
| 237 | `window.layoutTestsController` objects, which are synonyms. Reading the |
| 238 | `TestRunnerBindings::GetObjectTemplateBuilder` method tells us what properties |
| 239 | are available on the `window.testRunner` object. |
| 240 | |
| 241 | *** aside |
| 242 | `window.testRunner` is the preferred way to access the `testRunner` APIs. |
| 243 | `window.layoutTestsController` is still supported because it is used by |
| 244 | 3rd-party tests. |
| 245 | *** |
| 246 | |
| 247 | *** note |
| 248 | `testRunner` is the most popular testing API because it is also used indirectly |
| 249 | by tests that stick to Web Platform APIs. The `testharnessreport.js` file in |
| 250 | `testharness.js` is specifically designated to hold glue code that connects |
| 251 | `testharness.js` to the testing environment. Our implementation is in |
| 252 | [third_party/WebKit/LayoutTests/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js), |
| 253 | and uses the `testRunner` API. |
| 254 | *** |
| 255 | |
| 256 | See the [components/test_runner/](../../components/test_runner/) directory and |
| 257 | [WebKit's LayoutTests guide](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree) |
| 258 | for other useful APIs. For example, `window.eventSender` |
| 259 | ([components/test_runner/event_sender.h](../../components/test_runner/event_sender.h) |
| 260 | and |
| 261 | [components/test_runner/event_sender.cpp](../../components/test_runner/event_sender.cpp)) |
| 262 | has methods that simulate events input such as keyboard / mouse input and |
| 263 | drag-and-drop. |
| 264 | |
| 265 | Here is a UML diagram of how the `testRunner` bindings fit into Chromium. |
| 266 | |
| 267 | [](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit) |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 268 | |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 269 | ### Text Test Baselines |
| 270 | |
| 271 | By default, all the test cases in a file that uses `testharness.js` are expected |
| 272 | to pass. However, in some cases, we prefer to add failing test cases to the |
| 273 | repository, so that we can be notified when the failure modes change (e.g., we |
| 274 | want to know if a test starts crashing rather than returning incorrect output). |
| 275 | In these situations, a test file will be accompanied by a baseline, which is an |
| 276 | `-expected.txt` file that contains the test's expected output. |
| 277 | |
| 278 | The baselines are generated automatically when appropriate by |
| 279 | `run-webkit-tests`, which is described [here](./layout_tests.md), and by the |
| 280 | [rebaselining tools](./layout_test_expectations.md). |
| 281 | |
| 282 | Text baselines for `testharness.js` should be avoided, as having a text baseline |
| 283 | associated with a `testharness.js` indicates the presence of a bug. For this |
| 284 | reason, CLs that add text baselines must include a |
| 285 | [crbug.com](https://crbug.com) link for an issue tracking the removal of the |
| 286 | text expectations. |
| 287 | |
| 288 | * When creating tests that will be upstreamed to WPT, and Blink's current |
| 289 | behavior does not match the specification that is being tested, a text |
| 290 | baseline is necessary. Remember to create an issue tracking the expectation's |
| 291 | removal, and to link the issue in the CL description. |
| 292 | * Layout tests that cannot be upstreamed to WPT should use JavaScript to |
| 293 | document Blink's current behavior, rather than using JavaScript to document |
| 294 | desired behavior and a text file to document current behavior. |
| 295 | |
| 296 | ### The js-test.js Legacy Harness |
| 297 | |
| 298 | *** promo |
| 299 | For historical reasons, older tests are written using the `js-test` harness. |
| 300 | This harness is **deprecated**, and should not be used for new tests. |
| 301 | *** |
| 302 | |
| 303 | If you need to understand old tests, the best `js-test` documentation is its |
| 304 | implementation at |
| 305 | [third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/LayoutTests/resources/js-test.js). |
| 306 | |
| 307 | `js-test` tests lean heavily on the Blink-specific `testRunner` testing API. |
| 308 | In a nutshell, the tests call `testRunner.dumpAsText()` to signal that the page |
| 309 | content should be dumped and compared against a text baseline (an |
| 310 | `-expected.txt` file). As a consequence, `js-test` tests are always accompanied |
| 311 | by text baselines. Asynchronous tests also use `testRunner.waitUntilDone()` and |
| 312 | `testRunner.notifyDone()` to tell the testing tools when they are complete. |
| 313 | |
| 314 | ### Tests that use an HTTP Server |
| 315 | |
| 316 | By default, tests are loaded as if via `file:` URLs. Some web platform features |
| 317 | require tests served via HTTP or HTTPS, for example absolute paths (`src=/foo`) |
| 318 | or features restricted to secure protocols. |
| 319 | |
| 320 | HTTP tests are those under `LayoutTests/http/tests` (or virtual variants). Use a |
| 321 | locally running HTTP server (Apache) to run them. Tests are served off of ports |
| 322 | 8000 and 8080 for HTTP, and 8443 for HTTPS. If you run the tests using |
| 323 | `run-webkit-tests`, the server will be started automatically. To run the server |
| 324 | manually to reproduce or debug a failure: |
| 325 | |
| 326 | ```bash |
| 327 | cd src/third_party/WebKit/Tools/Scripts |
| 328 | run-blink-httpd start |
| 329 | ``` |
| 330 | |
| 331 | The layout tests will be served from `http://127.0.0.1:8000`. For example, to |
| 332 | run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`, |
| 333 | navigate to |
| 334 | `http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some |
| 335 | tests will behave differently if you go to 127.0.0.1 instead of localhost, so |
| 336 | use 127.0.0.1. |
| 337 | |
| 338 | To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill` |
| 339 | or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS. |
| 340 | |
| 341 | The test server sets up an alias to the `LayoutTests/resources` directory. In |
| 342 | HTTP tests, you can access the testing framework at e.g. |
pwnall | e781948 | 2016-12-17 00:58:40 | [diff] [blame] | 343 | `src="/resources/testharness.js"`. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 344 | |
| 345 | TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a |
| 346 | position to use it to run layout tests. |
| 347 | |
| 348 | ## Reference Tests (Reftests) |
| 349 | |
| 350 | Reference tests, also known as reftests, perform a pixel-by-pixel comparison |
| 351 | between the rendered image of a test page and the rendered image of a reference |
| 352 | page. Most reference tests pass if the two images match, but there are cases |
| 353 | where it is useful to have a test pass when the two images do _not_ match. |
| 354 | |
| 355 | Reference tests are more difficult to debug than JavaScript tests, and tend to |
| 356 | be slower as well. Therefore, they should only be used for functionality that |
| 357 | cannot be covered by JavaScript tests. |
| 358 | |
| 359 | New reference tests should follow the |
| 360 | [WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The |
| 361 | most important points are summarized below. |
| 362 | |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 363 | * 🚧 The test page declares the reference page using a |
| 364 | `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test |
| 365 | passes when the test image matches or does not match the reference image. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 366 | * The reference page must not use the feature being tested. Otherwise, the test |
| 367 | is meaningless. |
| 368 | * The reference page should be as simple as possible, and should not depend on |
| 369 | advanced features. Ideally, the reference page should render as intended even |
| 370 | on browsers with poor CSS support. |
| 371 | * Reference tests should be self-describing. |
| 372 | * Reference tests do _not_ include `testharness.js`. |
| 373 | |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 374 | 🚧 Our testing infrastructure was designed for the |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 375 | [WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink |
| 376 | has inherited. The consequences are summarized below. |
| 377 | |
| 378 | * Each reference page must be in the same directory as its associated test. |
| 379 | Given a test page named `foo` (e.g. `foo.html` or `foo.svg`), |
| 380 | * The reference page must be named `foo-expected` (e.g., |
| 381 | `foo-expected.html`) if the test passes when the two images match. |
| 382 | * The reference page must be named `foo-expected-mismatch` (e.g., |
| 383 | `foo-expected-mismatch.svg`) if the test passes when the two images do |
| 384 | _not_ match. |
| 385 | * Multiple references and chained references are not supported. |
| 386 | |
| 387 | The following example demonstrates a reference test for |
| 388 | [`<ol>`'s reversed attribute](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ol). |
| 389 | The example assumes that the test page is named `ol-reversed.html`. |
| 390 | |
| 391 | ```html |
| 392 | <!doctype html> |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 393 | <link rel="match" href="ol-reversed-expected.html"> |
| 394 | |
| 395 | <ol reversed> |
| 396 | <li>A</li> |
| 397 | <li>B</li> |
| 398 | <li>C</li> |
| 399 | </ol> |
| 400 | ``` |
| 401 | |
| 402 | The reference page, which must be named `ol-reversed-expected.html`, is below. |
| 403 | |
| 404 | ```html |
| 405 | <!doctype html> |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 406 | |
| 407 | <ol> |
| 408 | <li value="3">A</li> |
| 409 | <li value="2">B</li> |
| 410 | <li value="1">C</li> |
| 411 | </ol> |
| 412 | ``` |
| 413 | |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 414 | *** promo |
| 415 | The method for pointing out a test's reference page is still in flux, and is |
| 416 | being discussed on |
| 417 | [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion). |
| 418 | *** |
| 419 | |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 420 | ## Pixel Tests |
| 421 | |
| 422 | `testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and |
| 423 | `window.testRunner.dumpDragImage()` create an image result that is associated |
| 424 | with the test. The image result is compared against an image baseline, which is |
| 425 | an `-expected.png` file associated with the test, and the test passes if the |
| 426 | image result is identical to the baseline, according to a pixel-by-pixel |
| 427 | comparison. Tests that have image results (and baselines) are called **pixel |
| 428 | tests**. |
| 429 | |
| 430 | Pixel tests should still follow the principles laid out above. Pixel tests pose |
| 431 | unique challenges to the desire to have *self-describing* and *cross-platform* |
| 432 | tests. The |
| 433 | [WPT test style guidelines](http://testthewebforward.org/docs/test-style-guidelines.html) |
| 434 | contain useful guidance. The most relevant pieces of advice are below. |
| 435 | |
| 436 | * Whenever possible, use a green paragraph / page / square to indicate success. |
| 437 | If that is not possible, make the test self-describing by including a textual |
| 438 | description of the desired (passing) outcome. |
| 439 | * Only use the red color or the word `FAIL` to highlight errors. This does not |
| 440 | apply when testing the color red. |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 441 | * 🚧 Use the |
| 442 | [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the |
| 443 | variance introduced by the platform's text rendering system. This does not |
| 444 | apply when testing text, text flow, font selection, font fallback, font |
| 445 | features, or other typographic information. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 446 | |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 447 | TODO: Document how to opt out of generating a layout tree when generating |
| 448 | pixel results. |
| 449 | |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 450 | *** promo |
| 451 | When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result |
| 452 | will always be 800x600px, because test pages are rendered in an 800x600px |
| 453 | viewport. Pixel tests that do not specifically cover scrolling should fit in an |
| 454 | 800x600px viewport without creating scrollbars. |
| 455 | *** |
| 456 | |
pwnall | 6acacd8 | 2016-12-02 01:40:15 | [diff] [blame] | 457 | *** promo |
| 458 | The recommendation of using Ahem in pixel tests is being discussed on |
| 459 | [blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion). |
| 460 | *** |
| 461 | |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 462 | The following snippet includes the Ahem font in a layout test. |
| 463 | |
| 464 | ```html |
| 465 | <style> |
| 466 | body { |
| 467 | font: 10px Ahem; |
| 468 | } |
| 469 | </style> |
| 470 | <script src="/resources/ahem.js"></script> |
| 471 | ``` |
| 472 | |
| 473 | *** promo |
| 474 | Tests outside `LayoutTests/http` and `LayoutTests/imported/wpt` currently need |
| 475 | to use a relative path to |
| 476 | [/third_party/WebKit/LayoutTests/resources/ahem.js](../../third_party/WebKit/LayoutTests/resources/ahem.js) |
| 477 | *** |
| 478 | |
| 479 | ### Tests that need to paint, raster, or draw a frame of intermediate output |
| 480 | |
| 481 | A layout test does not actually draw frames of output until the test exits. |
| 482 | Tests that need to generate a painted frame can use |
| 483 | `window.testRunner.displayAsyncThen`, which will run the machinery to put up a |
| 484 | frame, then call the passed callback. There is also a library at |
| 485 | `fast/repaint/resources/text-based-repaint.js` to help with writing paint |
| 486 | invalidation and repaint tests. |
| 487 | |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 488 | ## Layout tree tests |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 489 | |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 490 | A layout tree test renders a web page and produces up to two results, which |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 491 | are compared against baseline files: |
| 492 | |
| 493 | * All tests output a textual representation of Blink's |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 494 | [layout tree](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/render-tree-construction) (called the render tree on that page), |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 495 | which is compared against an `-expected.txt` text baseline. |
| 496 | * Some tests also output the image of the rendered page, which is compared |
| 497 | against an `-expected.png` image baseline, using the same method as pixel |
| 498 | tests. |
| 499 | |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 500 | Whether you want a pixel test or a layout tree test depends on whether |
| 501 | you care about the visual image, the details of how that image was |
| 502 | constructed, or both. It is possible for multiple layout trees to produce |
| 503 | the same pixel output, so it is important to make it clear in the test |
| 504 | which outputs you really care about. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 505 | |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 506 | TODO: Document the API used by layout tree tests to opt out of producing image |
| 507 | results. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 508 | |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 509 | A layout tree test passes if _all_ of its results match their baselines. Like pixel |
| 510 | tests, the output of layout tree tests depends on platform-specific details, |
| 511 | so layout tree tests often require per-platform baselines. Furthermore, |
| 512 | since the tests obviously depend on the layout tree structure, |
| 513 | that means that if we change the layout tree you have to rebaseline each |
| 514 | layout tree test to see if the results are still correct and whether the test |
| 515 | is still meaningful. There are actually many cases where the layout tree |
| 516 | output is misstated (i.e., wrong), because people didn't want to have to update |
| 517 | existing baselines and tests. This is really unfortunate and confusing. |
| 518 | |
| 519 | For these reasons, layout tree tests should **only** be used to cover aspects |
| 520 | of the layout code that can only be tested by looking at the layout tree. Any |
| 521 | combination of the other test types is preferable to a layout tree test. |
| 522 | Layout tree tests are |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 523 | [inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 524 | the repository may have some unfortunate examples of layout tree tests. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 525 | |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 526 | |
| 527 | The following page is an example of a layout tree test. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 528 | |
| 529 | ```html |
| 530 | <!doctype html> |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 531 | <style> |
| 532 | body { font: 10px Ahem; } |
| 533 | span::after { |
| 534 | content: "pass"; |
| 535 | color: green; |
| 536 | } |
| 537 | </style> |
| 538 | <script src="/resources/ahem.js"></script> |
| 539 | |
| 540 | <p><span>Pass if a green PASS appears to the right: </span></p> |
| 541 | ``` |
| 542 | |
| 543 | The most important aspects of the example are that the test page does not |
| 544 | include a testing framework, and that it follows the guidelines for pixel tests. |
| 545 | The test page produces the text result below. |
| 546 | |
| 547 | ``` |
| 548 | layer at (0,0) size 800x600 |
| 549 | LayoutView at (0,0) size 800x600 |
| 550 | layer at (0,0) size 800x30 |
| 551 | LayoutBlockFlow {HTML} at (0,0) size 800x30 |
| 552 | LayoutBlockFlow {BODY} at (8,10) size 784x10 |
| 553 | LayoutBlockFlow {P} at (0,0) size 784x10 |
| 554 | LayoutInline {SPAN} at (0,0) size 470x10 |
| 555 | LayoutText {#text} at (0,0) size 430x10 |
| 556 | text run at (0,0) width 430: "Pass if a green PASS appears to the right: " |
| 557 | LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000] |
| 558 | LayoutTextFragment (anonymous) at (430,0) size 40x10 |
| 559 | text run at (430,0) width 40: "pass" |
| 560 | ``` |
| 561 | |
| 562 | Notice that the test result above depends on the size of the `<p>` text. The |
| 563 | test page uses the Ahem font (introduced above), whose main design goal is |
| 564 | consistent cross-platform rendering. Had the test used another font, its text |
| 565 | baseline would have depended on the fonts installed on the testing computer, and |
| 566 | on the platform's font rendering system. Please follow the pixel tests |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 567 | guidelines and write reliable layout tree tests! |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 568 | |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 569 | WebKit's layout tree is described in |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 570 | [a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/) |
dpranke | d2b7d64 | 2017-01-15 04:00:24 | [diff] [blame] | 571 | on WebKit's blog. Some of the concepts there still apply to Blink's layout tree. |
pwnall | 4ea2eb3 | 2016-11-29 02:47:25 | [diff] [blame] | 572 | |
| 573 | ## Directory Structure |
| 574 | |
| 575 | The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently |
| 576 | lacks a strict, formal structure. The following directories have special |
| 577 | meaning: |
| 578 | |
| 579 | * The `http/` directory hosts tests that require an HTTP server (see above). |
| 580 | * The `resources/` subdirectory in every directory contains binary files, such |
| 581 | as media files, and code that is shared by multiple test files. |
| 582 | |
| 583 | *** note |
| 584 | Some layout tests consist of a minimal HTML page that references a JavaScript |
| 585 | file in `resources/`. Please do not use this pattern for new tests, as it goes |
| 586 | against the minimality principle. JavaScript and CSS files should only live in |
| 587 | `resources/` if they are shared by at least two test files. |
| 588 | *** |