blob: 81146831e66ea7a3ad6bc5059302bc8f3ed608a2 [file] [log] [blame] [view]
pwnall4ea2eb32016-11-29 02:47:251# Writing Layout Tests
2
3_Layout tests_ is a bit of a misnomer. This term is
4[a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory/),
5and we use it to refer to every test that is written as a Web page (HTML, SVG,
6or XHTML) and lives in
7[third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8
9[TOC]
10
11## Overview
12
13Layout tests should be used to accomplish one of the following goals:
14
151. The entire surface of Blink that is exposed to the Web should be covered by
foolipeda32ab2017-02-16 19:21:5816 tests that we contribute to [web-platform-tests](./web_platform_tests.md)
17 (WPT). This helps us avoid regressions, and helps us identify Web Platform
18 areas where the major browsers don't have interoperable implementations.
19 Furthermore, by contributing to projects such as WPT, we share the burden of
20 writing tests with the other browser vendors, and we help all the browsers
21 get better. This is very much in line with our goal to move the Web forward.
pwnall4ea2eb32016-11-29 02:47:25222. When a Blink feature cannot be tested using the tools provided by WPT, and
23 cannot be easily covered by
24 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Source/web/tests/?q=webframetest&sq=package:chromium&type=cs),
25 the feature must be covered by layout tests, to avoid unexpected regressions.
26 These tests will use Blink-specific testing APIs that are only available in
27 [content_shell](./layout_tests_in_content_shell.md).
28
29*** promo
30If you know that Blink layout tests are upstreamed to other projects, such as
31[test262](https://github.com/tc39/test262), please update this document. Most
32importantly, our guidelines should to make it easy for our tests to be
pwnall6acacd82016-12-02 01:40:1533upstreamed. The
34[blink-dev mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/blink-dev)
35will be happy to help you harmonize our current guidelines with communal test
36repositories.
pwnall4ea2eb32016-11-29 02:47:2537***
38
39### Test Types
40
41There are four broad types of layout tests, listed in the order of preference.
42
43* *JavaScript Tests* are the layout test implementation of
44 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
45 assertions written in JavaScript, and pass if the assertions evaluate to
46 true.
47* *Reference Tests* render a test page and a reference page, and pass if the two
48 renderings are identical, according to a pixel-by-pixel comparison. These
49 tests are less robust, harder to debug, and significantly slower than
50 JavaScript tests, and are only used when JavaScript tests are insufficient,
51 such as when testing paint code.
52* *Pixel Tests* render a test page and compare the result against a pre-rendered
dpranked2b7d642017-01-15 04:00:2453 baseline image in the repository. Pixel tests are less robust than the
54 first two types, because the rendering of a page is influenced by
pwnall4ea2eb32016-11-29 02:47:2555 many factors such as the host computer's graphics card and driver, the
56 platform's text rendering system, and various user-configurable operating
57 system settings. For this reason, it is common for a pixel test to have a
dpranked2b7d642017-01-15 04:00:2458 different reference image for each platform that Blink is tested on, and
59 the reference images are
60 [quite cumbersome to manage](./layout_test_expectations.md). You
61 should only write a pixel test if you cannot use a reference test. By default
62 a pixel test will also dump the layout tree as text output, so they are
63 similar to ...
64* *Layout tree tests*, which output a textual representation of the layout
pwnall4ea2eb32016-11-29 02:47:2565 tree, which is the key data structure in Blink's page rendering system. The
dpranked2b7d642017-01-15 04:00:2466 test passes if the output matches a baseline text file in the repository.
67 Layout tree tests are used as a last resort to test the internal quirks of
68 the implementation, and they should be avoided in favor of one of the earlier
69 options.
pwnall59aadcb2017-01-26 23:27:2170
pwnall4ea2eb32016-11-29 02:47:2571## General Principles
72
pwnall6acacd82016-12-02 01:40:1573
74Tests should be written under the assumption that they will be upstreamed
pwnall59aadcb2017-01-26 23:27:2175to the WPT project. To this end, tests should follow the
foolipeda32ab2017-02-16 19:21:5876[WPT guidelines](http://web-platform-tests.org/writing-tests/).
pwnall6acacd82016-12-02 01:40:1577
pwnall6acacd82016-12-02 01:40:1578
pwnall59aadcb2017-01-26 23:27:2179There is no style guide that applies to all layout tests. However, some projects
80have adopted style guides, such as the
81[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
pwnall6acacd82016-12-02 01:40:1582
pwnall59aadcb2017-01-26 23:27:2183Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most
84important WPT guidelines and highlights some JavaScript concepts that are worth
85paying attention to when trying to infer style rules from existing tests. If
86you're unopinionated and looking for a style guide to follow, the document also
87suggests some defaults.
pwnall6acacd82016-12-02 01:40:1588
pwnall4ea2eb32016-11-29 02:47:2589## JavaScript Tests
90
91Whenever possible, the testing criteria should be expressed in JavaScript. The
92alternatives, which will be described in future sections, result in slower and
93less reliable tests.
94
95All new JavaScript tests should be written using the
96[testharness.js](https://github.com/w3c/testharness.js/) testing framework. This
97framework is used by the tests in the
98[web-platform-tests](https://github.com/w3c/web-platform-tests) repository,
99which is shared with all the other browser vendors, so `testharness.js` tests
100are more accessible to browser developers.
101
foolipeda32ab2017-02-16 19:21:58102See the [API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.md)
103for a thorough introduction to `testharness.js`.
104
105Layout tests should follow the recommendations of the above documentation.
pwnall4ea2eb32016-11-29 02:47:25106Furthermore, layout tests should include relevant
foolipeda32ab2017-02-16 19:21:58107[metadata](http://web-platform-tests.org/writing-tests/css-metadata.html). The
pwnall4ea2eb32016-11-29 02:47:25108specification URL (in `<link rel="help">`) is almost always relevant, and is
109incredibly helpful to a developer who needs to understand the test quickly.
110
111Below is a skeleton for a JavaScript test embedded in an HTML page. Note that,
112in order to follow the minimality guideline, the test omits the tags `<html>`,
113`<head>`, and `<body>`, as they can be inferred by the HTML parser.
114
115```html
116<!doctype html>
pwnall59aadcb2017-01-26 23:27:21117<title>JavaScript: the true literal is immutable and equal to itself</title>
pwnall4ea2eb32016-11-29 02:47:25118<link rel="help" href="https://tc39.github.io/ecma262/#sec-boolean-literals">
pwnall4ea2eb32016-11-29 02:47:25119<script src="/resources/testharness.js"></script>
120<script src="/resources/testharnessreport.js"></script>
121<script>
122'use strict';
123
124// Synchronous test example.
125test(() => {
126 const value = true;
127 assert_true(value, 'true literal');
128 assert_equals(value.toString(), 'true', 'the string representation of true');
129}, 'The literal true in a synchronous test case');
130
131// Asynchronous test example.
132async_test(t => {
133 const originallyTrue = true;
134 setTimeout(t.step_func_done(() => {
135 assert_equals(originallyTrue, true);
136 }), 0);
137}, 'The literal true in a setTimeout callback');
138
139// Promise test example.
140promise_test(() => {
141 return new Promise((resolve, reject) => {
142 resolve(true);
143 }).then(value => {
144 assert_true(value);
145 });
146}, 'The literal true used to resolve a Promise');
147
148</script>
149```
150
151Some points that are not immediately obvious from the example:
152
pwnall4ea2eb32016-11-29 02:47:25153* When calling an `assert_` function that compares two values, the first
154 argument is the actual value (produced by the functionality being tested), and
155 the second argument is the expected value (known good, golden). The order
156 is important, because the testing harness relies on it to generate expressive
157 error messages that are relied upon when debugging test failures.
158* The assertion description (the string argument to `assert_` methods) conveys
159 the way the actual value was obtained.
160 * If the expected value doesn't make it clear, the assertion description
161 should explain the desired behavior.
162 * Test cases with a single assertion should omit the assertion's description
163 when it is sufficiently clear.
164* Each test case describes the circumstance that it tests, without being
165 redundant.
166 * Do not start test case descriptions with redundant terms like "Testing"
167 or "Test for".
ktyliue0bb9882017-01-10 01:47:50168 * Test files with a single test case should omit the test case description.
169 The file's `<title>` should be sufficient to describe the scenario being
170 tested.
pwnall4ea2eb32016-11-29 02:47:25171* Asynchronous tests have a few subtleties.
172 * The `async_test` wrapper calls its function with a test case argument that
173 is used to signal when the test case is done, and to connect assertion
174 failures to the correct test.
175 * `t.done()` must be called after all the test case's assertions have
176 executed.
177 * Test case assertions (actually, any callback code that can throw
178 exceptions) must be wrapped in `t.step_func()` calls, so that
179 assertion failures and exceptions can be traced back to the correct test
180 case.
181 * `t.step_func_done()` is a shortcut that combines `t.step_func()` with a
182 `t.done()` call.
183
184*** promo
185Layout tests that load from `file://` origins must currently use relative paths
186to point to
187[/resources/testharness.js](../../third_party/WebKit/LayoutTests/resources/testharness.js)
188and
189[/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js).
190This is contrary to the WPT guidelines, which call for absolute paths.
191This limitation does not apply to the tests in `LayoutTests/http`, which rely on
foolip339204d2017-01-27 21:10:17192an HTTP server, or to the tests in `LayoutTests/external/wpt`, which are
pwnall4ea2eb32016-11-29 02:47:25193imported from the [WPT repository](https://github.com/w3c/web-platform-tests).
194***
195
196### WPT Supplemental Testing APIs
197
198Some tests simply cannot be expressed using the Web Platform APIs. For example,
199some tests that require a user to perform a gesture, such as a mouse click,
200cannot be implemented using Web APIs. The WPT project covers some of these cases
201via supplemental testing APIs.
202
pwnall59aadcb2017-01-26 23:27:21203When writing tests that rely on supplemental testing APIs, please consider the
204cost and benefits of having the tests
205[gracefully degrade to manual tests](./layout_tests_with_manual_fallback.md) in
206the absence of the testing APIs.
207
pwnall4ea2eb32016-11-29 02:47:25208*** promo
209In many cases, the user gesture is not actually necessary. For example, many
210event handling tests can use
211[synthetic events](https://developer.mozilla.org/docs/Web/Guide/Events/Creating_and_triggering_events).
212***
213
pwnall4ea2eb32016-11-29 02:47:25214### Relying on Blink-Specific Testing APIs
215
216Tests that cannot be expressed using the Web Platform APIs or WPT's testing APIs
217use Blink-specific testing APIs. These APIs are only available in
218[content_shell](./layout_tests_in_content_shell.md), and should only be used as
219a last resort.
220
221A downside of Blink-specific APIs is that they are not as well documented as the
222Web Platform features. Learning to use a Blink-specific feature requires finding
223other tests that use it, or reading its source code.
224
225For example, the most popular Blink-specific API is `testRunner`, which is
226implemented in
227[components/test_runner/test_runner.h](../../components/test_runner/test_runner.h)
228and
qyearsley23599b72017-02-16 19:10:42229[components/test_runner/test_runner.cc](../../components/test_runner/test_runner.cc).
pwnall4ea2eb32016-11-29 02:47:25230By skimming the `TestRunnerBindings::Install` method, we learn that the
231testRunner API is presented by the `window.testRunner` and
232`window.layoutTestsController` objects, which are synonyms. Reading the
233`TestRunnerBindings::GetObjectTemplateBuilder` method tells us what properties
234are available on the `window.testRunner` object.
235
236*** aside
237`window.testRunner` is the preferred way to access the `testRunner` APIs.
238`window.layoutTestsController` is still supported because it is used by
2393rd-party tests.
240***
241
242*** note
243`testRunner` is the most popular testing API because it is also used indirectly
244by tests that stick to Web Platform APIs. The `testharnessreport.js` file in
245`testharness.js` is specifically designated to hold glue code that connects
246`testharness.js` to the testing environment. Our implementation is in
247[third_party/WebKit/LayoutTests/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js),
248and uses the `testRunner` API.
249***
250
251See the [components/test_runner/](../../components/test_runner/) directory and
252[WebKit's LayoutTests guide](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree)
253for other useful APIs. For example, `window.eventSender`
254([components/test_runner/event_sender.h](../../components/test_runner/event_sender.h)
255and
qyearsley23599b72017-02-16 19:10:42256[components/test_runner/event_sender.cc](../../components/test_runner/event_sender.cc))
pwnall4ea2eb32016-11-29 02:47:25257has methods that simulate events input such as keyboard / mouse input and
258drag-and-drop.
259
260Here is a UML diagram of how the `testRunner` bindings fit into Chromium.
261
262[![UML of testRunner bindings configuring platform implementation](https://docs.google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/svg?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
pwnall6acacd82016-12-02 01:40:15263
pwnall4ea2eb32016-11-29 02:47:25264### Text Test Baselines
265
266By default, all the test cases in a file that uses `testharness.js` are expected
267to pass. However, in some cases, we prefer to add failing test cases to the
268repository, so that we can be notified when the failure modes change (e.g., we
269want to know if a test starts crashing rather than returning incorrect output).
270In these situations, a test file will be accompanied by a baseline, which is an
271`-expected.txt` file that contains the test's expected output.
272
273The baselines are generated automatically when appropriate by
274`run-webkit-tests`, which is described [here](./layout_tests.md), and by the
275[rebaselining tools](./layout_test_expectations.md).
276
277Text baselines for `testharness.js` should be avoided, as having a text baseline
278associated with a `testharness.js` indicates the presence of a bug. For this
279reason, CLs that add text baselines must include a
280[crbug.com](https://crbug.com) link for an issue tracking the removal of the
281text expectations.
282
283* When creating tests that will be upstreamed to WPT, and Blink's current
284 behavior does not match the specification that is being tested, a text
285 baseline is necessary. Remember to create an issue tracking the expectation's
286 removal, and to link the issue in the CL description.
287* Layout tests that cannot be upstreamed to WPT should use JavaScript to
288 document Blink's current behavior, rather than using JavaScript to document
289 desired behavior and a text file to document current behavior.
290
291### The js-test.js Legacy Harness
292
293*** promo
294For historical reasons, older tests are written using the `js-test` harness.
295This harness is **deprecated**, and should not be used for new tests.
296***
297
298If you need to understand old tests, the best `js-test` documentation is its
299implementation at
300[third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/LayoutTests/resources/js-test.js).
301
302`js-test` tests lean heavily on the Blink-specific `testRunner` testing API.
303In a nutshell, the tests call `testRunner.dumpAsText()` to signal that the page
304content should be dumped and compared against a text baseline (an
305`-expected.txt` file). As a consequence, `js-test` tests are always accompanied
306by text baselines. Asynchronous tests also use `testRunner.waitUntilDone()` and
307`testRunner.notifyDone()` to tell the testing tools when they are complete.
308
309### Tests that use an HTTP Server
310
311By default, tests are loaded as if via `file:` URLs. Some web platform features
312require tests served via HTTP or HTTPS, for example absolute paths (`src=/foo`)
313or features restricted to secure protocols.
314
315HTTP tests are those under `LayoutTests/http/tests` (or virtual variants). Use a
316locally running HTTP server (Apache) to run them. Tests are served off of ports
3178000 and 8080 for HTTP, and 8443 for HTTPS. If you run the tests using
318`run-webkit-tests`, the server will be started automatically. To run the server
319manually to reproduce or debug a failure:
320
321```bash
322cd src/third_party/WebKit/Tools/Scripts
323run-blink-httpd start
324```
325
326The layout tests will be served from `http://127.0.0.1:8000`. For example, to
327run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
328navigate to
329`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
330tests will behave differently if you go to 127.0.0.1 instead of localhost, so
331use 127.0.0.1.
332
333To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill`
334or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS.
335
336The test server sets up an alias to the `LayoutTests/resources` directory. In
337HTTP tests, you can access the testing framework at e.g.
pwnalle7819482016-12-17 00:58:40338`src="/resources/testharness.js"`.
pwnall4ea2eb32016-11-29 02:47:25339
340TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
341position to use it to run layout tests.
342
343## Reference Tests (Reftests)
344
345Reference tests, also known as reftests, perform a pixel-by-pixel comparison
346between the rendered image of a test page and the rendered image of a reference
347page. Most reference tests pass if the two images match, but there are cases
348where it is useful to have a test pass when the two images do _not_ match.
349
350Reference tests are more difficult to debug than JavaScript tests, and tend to
351be slower as well. Therefore, they should only be used for functionality that
352cannot be covered by JavaScript tests.
353
354New reference tests should follow the
foolipeda32ab2017-02-16 19:21:58355[WPT reftests guidelines](http://web-platform-tests.org/writing-tests/reftests.html).
356The most important points are summarized below.
pwnall4ea2eb32016-11-29 02:47:25357
pwnall6acacd82016-12-02 01:40:15358* &#x1F6A7; The test page declares the reference page using a
359 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test
360 passes when the test image matches or does not match the reference image.
pwnall4ea2eb32016-11-29 02:47:25361* The reference page must not use the feature being tested. Otherwise, the test
362 is meaningless.
363* The reference page should be as simple as possible, and should not depend on
364 advanced features. Ideally, the reference page should render as intended even
365 on browsers with poor CSS support.
366* Reference tests should be self-describing.
367* Reference tests do _not_ include `testharness.js`.
368
pwnall6acacd82016-12-02 01:40:15369&#x1F6A7; Our testing infrastructure was designed for the
pwnall4ea2eb32016-11-29 02:47:25370[WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink
371has inherited. The consequences are summarized below.
372
373* Each reference page must be in the same directory as its associated test.
374 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`),
375 * The reference page must be named `foo-expected` (e.g.,
376 `foo-expected.html`) if the test passes when the two images match.
377 * The reference page must be named `foo-expected-mismatch` (e.g.,
378 `foo-expected-mismatch.svg`) if the test passes when the two images do
379 _not_ match.
380* Multiple references and chained references are not supported.
381
382The following example demonstrates a reference test for
383[`<ol>`'s reversed attribute](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ol).
384The example assumes that the test page is named `ol-reversed.html`.
385
386```html
387<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25388<link rel="match" href="ol-reversed-expected.html">
389
390<ol reversed>
391 <li>A</li>
392 <li>B</li>
393 <li>C</li>
394</ol>
395```
396
397The reference page, which must be named `ol-reversed-expected.html`, is below.
398
399```html
400<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25401
402<ol>
403 <li value="3">A</li>
404 <li value="2">B</li>
405 <li value="1">C</li>
406</ol>
407```
408
pwnall6acacd82016-12-02 01:40:15409*** promo
410The method for pointing out a test's reference page is still in flux, and is
411being discussed on
412[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
413***
414
pwnall4ea2eb32016-11-29 02:47:25415## Pixel Tests
416
417`testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and
418`window.testRunner.dumpDragImage()` create an image result that is associated
419with the test. The image result is compared against an image baseline, which is
420an `-expected.png` file associated with the test, and the test passes if the
421image result is identical to the baseline, according to a pixel-by-pixel
422comparison. Tests that have image results (and baselines) are called **pixel
423tests**.
424
425Pixel tests should still follow the principles laid out above. Pixel tests pose
426unique challenges to the desire to have *self-describing* and *cross-platform*
427tests. The
foolipeda32ab2017-02-16 19:21:58428[WPT rendering test guidelines](http://web-platform-tests.org/writing-tests/rendering.html)
pwnall4ea2eb32016-11-29 02:47:25429contain useful guidance. The most relevant pieces of advice are below.
430
431* Whenever possible, use a green paragraph / page / square to indicate success.
432 If that is not possible, make the test self-describing by including a textual
433 description of the desired (passing) outcome.
434* Only use the red color or the word `FAIL` to highlight errors. This does not
435 apply when testing the color red.
pwnall6acacd82016-12-02 01:40:15436* &#x1F6A7; Use the
437 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the
438 variance introduced by the platform's text rendering system. This does not
439 apply when testing text, text flow, font selection, font fallback, font
440 features, or other typographic information.
pwnall4ea2eb32016-11-29 02:47:25441
dpranked2b7d642017-01-15 04:00:24442TODO: Document how to opt out of generating a layout tree when generating
443pixel results.
444
pwnall4ea2eb32016-11-29 02:47:25445*** promo
446When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result
447will always be 800x600px, because test pages are rendered in an 800x600px
448viewport. Pixel tests that do not specifically cover scrolling should fit in an
449800x600px viewport without creating scrollbars.
450***
451
pwnall6acacd82016-12-02 01:40:15452*** promo
453The recommendation of using Ahem in pixel tests is being discussed on
454[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
455***
456
pwnall4ea2eb32016-11-29 02:47:25457The following snippet includes the Ahem font in a layout test.
458
459```html
460<style>
461body {
462 font: 10px Ahem;
463}
464</style>
465<script src="/resources/ahem.js"></script>
466```
467
468*** promo
foolip339204d2017-01-27 21:10:17469Tests outside `LayoutTests/http` and `LayoutTests/external/wpt` currently need
pwnall4ea2eb32016-11-29 02:47:25470to use a relative path to
471[/third_party/WebKit/LayoutTests/resources/ahem.js](../../third_party/WebKit/LayoutTests/resources/ahem.js)
472***
473
474### Tests that need to paint, raster, or draw a frame of intermediate output
475
476A layout test does not actually draw frames of output until the test exits.
477Tests that need to generate a painted frame can use
478`window.testRunner.displayAsyncThen`, which will run the machinery to put up a
479frame, then call the passed callback. There is also a library at
480`fast/repaint/resources/text-based-repaint.js` to help with writing paint
481invalidation and repaint tests.
482
dpranked2b7d642017-01-15 04:00:24483## Layout tree tests
pwnall4ea2eb32016-11-29 02:47:25484
dpranked2b7d642017-01-15 04:00:24485A layout tree test renders a web page and produces up to two results, which
pwnall4ea2eb32016-11-29 02:47:25486are compared against baseline files:
487
488* All tests output a textual representation of Blink's
dpranked2b7d642017-01-15 04:00:24489 [layout tree](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/render-tree-construction) (called the render tree on that page),
pwnall4ea2eb32016-11-29 02:47:25490 which is compared against an `-expected.txt` text baseline.
491* Some tests also output the image of the rendered page, which is compared
492 against an `-expected.png` image baseline, using the same method as pixel
493 tests.
494
dpranked2b7d642017-01-15 04:00:24495Whether you want a pixel test or a layout tree test depends on whether
496you care about the visual image, the details of how that image was
497constructed, or both. It is possible for multiple layout trees to produce
498the same pixel output, so it is important to make it clear in the test
499which outputs you really care about.
pwnall4ea2eb32016-11-29 02:47:25500
dpranked2b7d642017-01-15 04:00:24501TODO: Document the API used by layout tree tests to opt out of producing image
502results.
pwnall4ea2eb32016-11-29 02:47:25503
dpranked2b7d642017-01-15 04:00:24504A layout tree test passes if _all_ of its results match their baselines. Like pixel
505tests, the output of layout tree tests depends on platform-specific details,
506so layout tree tests often require per-platform baselines. Furthermore,
507since the tests obviously depend on the layout tree structure,
508that means that if we change the layout tree you have to rebaseline each
509layout tree test to see if the results are still correct and whether the test
510is still meaningful. There are actually many cases where the layout tree
511output is misstated (i.e., wrong), because people didn't want to have to update
512existing baselines and tests. This is really unfortunate and confusing.
513
514For these reasons, layout tree tests should **only** be used to cover aspects
515of the layout code that can only be tested by looking at the layout tree. Any
516combination of the other test types is preferable to a layout tree test.
517Layout tree tests are
pwnall4ea2eb32016-11-29 02:47:25518[inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so
dpranked2b7d642017-01-15 04:00:24519the repository may have some unfortunate examples of layout tree tests.
pwnall4ea2eb32016-11-29 02:47:25520
dpranked2b7d642017-01-15 04:00:24521
522The following page is an example of a layout tree test.
pwnall4ea2eb32016-11-29 02:47:25523
524```html
525<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25526<style>
527body { font: 10px Ahem; }
528span::after {
529 content: "pass";
530 color: green;
531}
532</style>
533<script src="/resources/ahem.js"></script>
534
535<p><span>Pass if a green PASS appears to the right: </span></p>
536```
537
538The most important aspects of the example are that the test page does not
539include a testing framework, and that it follows the guidelines for pixel tests.
540The test page produces the text result below.
541
542```
543layer at (0,0) size 800x600
544 LayoutView at (0,0) size 800x600
545layer at (0,0) size 800x30
546 LayoutBlockFlow {HTML} at (0,0) size 800x30
547 LayoutBlockFlow {BODY} at (8,10) size 784x10
548 LayoutBlockFlow {P} at (0,0) size 784x10
549 LayoutInline {SPAN} at (0,0) size 470x10
550 LayoutText {#text} at (0,0) size 430x10
551 text run at (0,0) width 430: "Pass if a green PASS appears to the right: "
552 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000]
553 LayoutTextFragment (anonymous) at (430,0) size 40x10
554 text run at (430,0) width 40: "pass"
555```
556
557Notice that the test result above depends on the size of the `<p>` text. The
558test page uses the Ahem font (introduced above), whose main design goal is
559consistent cross-platform rendering. Had the test used another font, its text
560baseline would have depended on the fonts installed on the testing computer, and
561on the platform's font rendering system. Please follow the pixel tests
dpranked2b7d642017-01-15 04:00:24562guidelines and write reliable layout tree tests!
pwnall4ea2eb32016-11-29 02:47:25563
dpranked2b7d642017-01-15 04:00:24564WebKit's layout tree is described in
pwnall4ea2eb32016-11-29 02:47:25565[a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/)
dpranked2b7d642017-01-15 04:00:24566on WebKit's blog. Some of the concepts there still apply to Blink's layout tree.
pwnall4ea2eb32016-11-29 02:47:25567
568## Directory Structure
569
570The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently
571lacks a strict, formal structure. The following directories have special
572meaning:
573
574* The `http/` directory hosts tests that require an HTTP server (see above).
575* The `resources/` subdirectory in every directory contains binary files, such
576 as media files, and code that is shared by multiple test files.
577
578*** note
579Some layout tests consist of a minimal HTML page that references a JavaScript
580file in `resources/`. Please do not use this pattern for new tests, as it goes
581against the minimality principle. JavaScript and CSS files should only live in
582`resources/` if they are shared by at least two test files.
583***