blob: d8e6f8ee112fcb621c3a3fe0c70c8826761deddb [file] [log] [blame] [view]
pwnall4ea2eb32016-11-29 02:47:251# Writing Layout Tests
2
3_Layout tests_ is a bit of a misnomer. This term is
4[a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory/),
5and we use it to refer to every test that is written as a Web page (HTML, SVG,
6or XHTML) and lives in
7[third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8
9[TOC]
10
11## Overview
12
13Layout tests should be used to accomplish one of the following goals:
14
151. The entire surface of Blink that is exposed to the Web should be covered by
foolipeda32ab2017-02-16 19:21:5816 tests that we contribute to [web-platform-tests](./web_platform_tests.md)
17 (WPT). This helps us avoid regressions, and helps us identify Web Platform
18 areas where the major browsers don't have interoperable implementations.
19 Furthermore, by contributing to projects such as WPT, we share the burden of
20 writing tests with the other browser vendors, and we help all the browsers
21 get better. This is very much in line with our goal to move the Web forward.
pwnall4ea2eb32016-11-29 02:47:25222. When a Blink feature cannot be tested using the tools provided by WPT, and
23 cannot be easily covered by
24 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Source/web/tests/?q=webframetest&sq=package:chromium&type=cs),
25 the feature must be covered by layout tests, to avoid unexpected regressions.
26 These tests will use Blink-specific testing APIs that are only available in
27 [content_shell](./layout_tests_in_content_shell.md).
28
29*** promo
30If you know that Blink layout tests are upstreamed to other projects, such as
31[test262](https://github.com/tc39/test262), please update this document. Most
32importantly, our guidelines should to make it easy for our tests to be
pwnall6acacd82016-12-02 01:40:1533upstreamed. The
34[blink-dev mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/blink-dev)
35will be happy to help you harmonize our current guidelines with communal test
36repositories.
pwnall4ea2eb32016-11-29 02:47:2537***
38
39### Test Types
40
41There are four broad types of layout tests, listed in the order of preference.
42
43* *JavaScript Tests* are the layout test implementation of
44 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
45 assertions written in JavaScript, and pass if the assertions evaluate to
46 true.
47* *Reference Tests* render a test page and a reference page, and pass if the two
48 renderings are identical, according to a pixel-by-pixel comparison. These
49 tests are less robust, harder to debug, and significantly slower than
50 JavaScript tests, and are only used when JavaScript tests are insufficient,
51 such as when testing paint code.
52* *Pixel Tests* render a test page and compare the result against a pre-rendered
dpranked2b7d642017-01-15 04:00:2453 baseline image in the repository. Pixel tests are less robust than the
54 first two types, because the rendering of a page is influenced by
pwnall4ea2eb32016-11-29 02:47:2555 many factors such as the host computer's graphics card and driver, the
56 platform's text rendering system, and various user-configurable operating
57 system settings. For this reason, it is common for a pixel test to have a
dpranked2b7d642017-01-15 04:00:2458 different reference image for each platform that Blink is tested on, and
59 the reference images are
60 [quite cumbersome to manage](./layout_test_expectations.md). You
61 should only write a pixel test if you cannot use a reference test. By default
62 a pixel test will also dump the layout tree as text output, so they are
63 similar to ...
64* *Layout tree tests*, which output a textual representation of the layout
pwnall4ea2eb32016-11-29 02:47:2565 tree, which is the key data structure in Blink's page rendering system. The
dpranked2b7d642017-01-15 04:00:2466 test passes if the output matches a baseline text file in the repository.
67 Layout tree tests are used as a last resort to test the internal quirks of
68 the implementation, and they should be avoided in favor of one of the earlier
69 options.
pwnall59aadcb2017-01-26 23:27:2170
pwnall4ea2eb32016-11-29 02:47:2571## General Principles
72
pwnall6acacd82016-12-02 01:40:1573
74Tests should be written under the assumption that they will be upstreamed
pwnall59aadcb2017-01-26 23:27:2175to the WPT project. To this end, tests should follow the
foolipeda32ab2017-02-16 19:21:5876[WPT guidelines](http://web-platform-tests.org/writing-tests/).
pwnall6acacd82016-12-02 01:40:1577
pwnall6acacd82016-12-02 01:40:1578
pwnall59aadcb2017-01-26 23:27:2179There is no style guide that applies to all layout tests. However, some projects
80have adopted style guides, such as the
81[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
pwnall6acacd82016-12-02 01:40:1582
pwnall59aadcb2017-01-26 23:27:2183Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most
84important WPT guidelines and highlights some JavaScript concepts that are worth
85paying attention to when trying to infer style rules from existing tests. If
86you're unopinionated and looking for a style guide to follow, the document also
87suggests some defaults.
pwnall6acacd82016-12-02 01:40:1588
pwnall4ea2eb32016-11-29 02:47:2589## JavaScript Tests
90
91Whenever possible, the testing criteria should be expressed in JavaScript. The
92alternatives, which will be described in future sections, result in slower and
93less reliable tests.
94
95All new JavaScript tests should be written using the
mekb330e312017-05-03 19:56:2296[testharness.js](https://github.com/w3c/web-platform-tests/tree/master/resources)
97testing framework. This framework is used by the tests in the
pwnall4ea2eb32016-11-29 02:47:2598[web-platform-tests](https://github.com/w3c/web-platform-tests) repository,
99which is shared with all the other browser vendors, so `testharness.js` tests
100are more accessible to browser developers.
101
mekb330e312017-05-03 19:56:22102See the [API documentation](http://web-platform-tests.org/writing-tests/testharness-api.html)
foolipeda32ab2017-02-16 19:21:58103for a thorough introduction to `testharness.js`.
104
105Layout tests should follow the recommendations of the above documentation.
pwnall4ea2eb32016-11-29 02:47:25106Furthermore, layout tests should include relevant
foolipeda32ab2017-02-16 19:21:58107[metadata](http://web-platform-tests.org/writing-tests/css-metadata.html). The
pwnall4ea2eb32016-11-29 02:47:25108specification URL (in `<link rel="help">`) is almost always relevant, and is
109incredibly helpful to a developer who needs to understand the test quickly.
110
111Below is a skeleton for a JavaScript test embedded in an HTML page. Note that,
112in order to follow the minimality guideline, the test omits the tags `<html>`,
113`<head>`, and `<body>`, as they can be inferred by the HTML parser.
114
115```html
116<!doctype html>
pwnall59aadcb2017-01-26 23:27:21117<title>JavaScript: the true literal is immutable and equal to itself</title>
pwnall4ea2eb32016-11-29 02:47:25118<link rel="help" href="https://tc39.github.io/ecma262/#sec-boolean-literals">
pwnall4ea2eb32016-11-29 02:47:25119<script src="/resources/testharness.js"></script>
120<script src="/resources/testharnessreport.js"></script>
121<script>
122'use strict';
123
124// Synchronous test example.
125test(() => {
126 const value = true;
127 assert_true(value, 'true literal');
128 assert_equals(value.toString(), 'true', 'the string representation of true');
129}, 'The literal true in a synchronous test case');
130
131// Asynchronous test example.
132async_test(t => {
133 const originallyTrue = true;
134 setTimeout(t.step_func_done(() => {
135 assert_equals(originallyTrue, true);
136 }), 0);
137}, 'The literal true in a setTimeout callback');
138
139// Promise test example.
140promise_test(() => {
141 return new Promise((resolve, reject) => {
142 resolve(true);
143 }).then(value => {
144 assert_true(value);
145 });
146}, 'The literal true used to resolve a Promise');
147
148</script>
149```
150
151Some points that are not immediately obvious from the example:
152
pwnall4ea2eb32016-11-29 02:47:25153* When calling an `assert_` function that compares two values, the first
154 argument is the actual value (produced by the functionality being tested), and
155 the second argument is the expected value (known good, golden). The order
156 is important, because the testing harness relies on it to generate expressive
157 error messages that are relied upon when debugging test failures.
158* The assertion description (the string argument to `assert_` methods) conveys
159 the way the actual value was obtained.
160 * If the expected value doesn't make it clear, the assertion description
161 should explain the desired behavior.
162 * Test cases with a single assertion should omit the assertion's description
163 when it is sufficiently clear.
164* Each test case describes the circumstance that it tests, without being
165 redundant.
166 * Do not start test case descriptions with redundant terms like "Testing"
167 or "Test for".
ktyliue0bb9882017-01-10 01:47:50168 * Test files with a single test case should omit the test case description.
169 The file's `<title>` should be sufficient to describe the scenario being
170 tested.
pwnall4ea2eb32016-11-29 02:47:25171* Asynchronous tests have a few subtleties.
172 * The `async_test` wrapper calls its function with a test case argument that
173 is used to signal when the test case is done, and to connect assertion
174 failures to the correct test.
175 * `t.done()` must be called after all the test case's assertions have
176 executed.
177 * Test case assertions (actually, any callback code that can throw
178 exceptions) must be wrapped in `t.step_func()` calls, so that
179 assertion failures and exceptions can be traced back to the correct test
180 case.
181 * `t.step_func_done()` is a shortcut that combines `t.step_func()` with a
182 `t.done()` call.
183
184*** promo
185Layout tests that load from `file://` origins must currently use relative paths
186to point to
187[/resources/testharness.js](../../third_party/WebKit/LayoutTests/resources/testharness.js)
188and
189[/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js).
190This is contrary to the WPT guidelines, which call for absolute paths.
191This limitation does not apply to the tests in `LayoutTests/http`, which rely on
foolip339204d2017-01-27 21:10:17192an HTTP server, or to the tests in `LayoutTests/external/wpt`, which are
pwnall4ea2eb32016-11-29 02:47:25193imported from the [WPT repository](https://github.com/w3c/web-platform-tests).
194***
195
196### WPT Supplemental Testing APIs
197
198Some tests simply cannot be expressed using the Web Platform APIs. For example,
199some tests that require a user to perform a gesture, such as a mouse click,
200cannot be implemented using Web APIs. The WPT project covers some of these cases
201via supplemental testing APIs.
202
pwnall59aadcb2017-01-26 23:27:21203When writing tests that rely on supplemental testing APIs, please consider the
204cost and benefits of having the tests
205[gracefully degrade to manual tests](./layout_tests_with_manual_fallback.md) in
206the absence of the testing APIs.
207
pwnall4ea2eb32016-11-29 02:47:25208*** promo
209In many cases, the user gesture is not actually necessary. For example, many
210event handling tests can use
211[synthetic events](https://developer.mozilla.org/docs/Web/Guide/Events/Creating_and_triggering_events).
212***
213
pwnall4ea2eb32016-11-29 02:47:25214### Relying on Blink-Specific Testing APIs
215
216Tests that cannot be expressed using the Web Platform APIs or WPT's testing APIs
217use Blink-specific testing APIs. These APIs are only available in
218[content_shell](./layout_tests_in_content_shell.md), and should only be used as
219a last resort.
220
221A downside of Blink-specific APIs is that they are not as well documented as the
222Web Platform features. Learning to use a Blink-specific feature requires finding
223other tests that use it, or reading its source code.
224
225For example, the most popular Blink-specific API is `testRunner`, which is
226implemented in
227[components/test_runner/test_runner.h](../../components/test_runner/test_runner.h)
228and
qyearsley23599b72017-02-16 19:10:42229[components/test_runner/test_runner.cc](../../components/test_runner/test_runner.cc).
pwnall4ea2eb32016-11-29 02:47:25230By skimming the `TestRunnerBindings::Install` method, we learn that the
231testRunner API is presented by the `window.testRunner` and
232`window.layoutTestsController` objects, which are synonyms. Reading the
233`TestRunnerBindings::GetObjectTemplateBuilder` method tells us what properties
234are available on the `window.testRunner` object.
235
236*** aside
237`window.testRunner` is the preferred way to access the `testRunner` APIs.
238`window.layoutTestsController` is still supported because it is used by
2393rd-party tests.
240***
241
242*** note
243`testRunner` is the most popular testing API because it is also used indirectly
244by tests that stick to Web Platform APIs. The `testharnessreport.js` file in
245`testharness.js` is specifically designated to hold glue code that connects
246`testharness.js` to the testing environment. Our implementation is in
247[third_party/WebKit/LayoutTests/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js),
248and uses the `testRunner` API.
249***
250
251See the [components/test_runner/](../../components/test_runner/) directory and
252[WebKit's LayoutTests guide](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree)
253for other useful APIs. For example, `window.eventSender`
254([components/test_runner/event_sender.h](../../components/test_runner/event_sender.h)
255and
qyearsley23599b72017-02-16 19:10:42256[components/test_runner/event_sender.cc](../../components/test_runner/event_sender.cc))
pwnall4ea2eb32016-11-29 02:47:25257has methods that simulate events input such as keyboard / mouse input and
258drag-and-drop.
259
260Here is a UML diagram of how the `testRunner` bindings fit into Chromium.
261
262[![UML of testRunner bindings configuring platform implementation](https://docs.google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/svg?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
pwnall6acacd82016-12-02 01:40:15263
pwnall4ea2eb32016-11-29 02:47:25264### Text Test Baselines
265
266By default, all the test cases in a file that uses `testharness.js` are expected
267to pass. However, in some cases, we prefer to add failing test cases to the
268repository, so that we can be notified when the failure modes change (e.g., we
269want to know if a test starts crashing rather than returning incorrect output).
270In these situations, a test file will be accompanied by a baseline, which is an
271`-expected.txt` file that contains the test's expected output.
272
273The baselines are generated automatically when appropriate by
274`run-webkit-tests`, which is described [here](./layout_tests.md), and by the
275[rebaselining tools](./layout_test_expectations.md).
276
277Text baselines for `testharness.js` should be avoided, as having a text baseline
278associated with a `testharness.js` indicates the presence of a bug. For this
279reason, CLs that add text baselines must include a
280[crbug.com](https://crbug.com) link for an issue tracking the removal of the
281text expectations.
282
283* When creating tests that will be upstreamed to WPT, and Blink's current
284 behavior does not match the specification that is being tested, a text
285 baseline is necessary. Remember to create an issue tracking the expectation's
286 removal, and to link the issue in the CL description.
287* Layout tests that cannot be upstreamed to WPT should use JavaScript to
288 document Blink's current behavior, rather than using JavaScript to document
289 desired behavior and a text file to document current behavior.
290
291### The js-test.js Legacy Harness
292
293*** promo
294For historical reasons, older tests are written using the `js-test` harness.
295This harness is **deprecated**, and should not be used for new tests.
296***
297
298If you need to understand old tests, the best `js-test` documentation is its
299implementation at
300[third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/LayoutTests/resources/js-test.js).
301
302`js-test` tests lean heavily on the Blink-specific `testRunner` testing API.
303In a nutshell, the tests call `testRunner.dumpAsText()` to signal that the page
304content should be dumped and compared against a text baseline (an
305`-expected.txt` file). As a consequence, `js-test` tests are always accompanied
306by text baselines. Asynchronous tests also use `testRunner.waitUntilDone()` and
307`testRunner.notifyDone()` to tell the testing tools when they are complete.
308
309### Tests that use an HTTP Server
310
311By default, tests are loaded as if via `file:` URLs. Some web platform features
312require tests served via HTTP or HTTPS, for example absolute paths (`src=/foo`)
313or features restricted to secure protocols.
314
315HTTP tests are those under `LayoutTests/http/tests` (or virtual variants). Use a
316locally running HTTP server (Apache) to run them. Tests are served off of ports
3178000 and 8080 for HTTP, and 8443 for HTTPS. If you run the tests using
318`run-webkit-tests`, the server will be started automatically. To run the server
319manually to reproduce or debug a failure:
320
321```bash
322cd src/third_party/WebKit/Tools/Scripts
Hajime Hoshia6fad022017-08-01 17:57:58323./run-blink-httpd
pwnall4ea2eb32016-11-29 02:47:25324```
325
326The layout tests will be served from `http://127.0.0.1:8000`. For example, to
327run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
328navigate to
329`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
330tests will behave differently if you go to 127.0.0.1 instead of localhost, so
331use 127.0.0.1.
332
Hajime Hoshia6fad022017-08-01 17:57:58333To kill the server, hit any key on the terminal where `run-blink-httpd` is
334running, or just use `taskkill` or the Task Manager on Windows, and `killall` or
335Activity Monitor on MacOS.
pwnall4ea2eb32016-11-29 02:47:25336
337The test server sets up an alias to the `LayoutTests/resources` directory. In
338HTTP tests, you can access the testing framework at e.g.
pwnalle7819482016-12-17 00:58:40339`src="/resources/testharness.js"`.
pwnall4ea2eb32016-11-29 02:47:25340
341TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
342position to use it to run layout tests.
343
344## Reference Tests (Reftests)
345
346Reference tests, also known as reftests, perform a pixel-by-pixel comparison
347between the rendered image of a test page and the rendered image of a reference
348page. Most reference tests pass if the two images match, but there are cases
349where it is useful to have a test pass when the two images do _not_ match.
350
351Reference tests are more difficult to debug than JavaScript tests, and tend to
352be slower as well. Therefore, they should only be used for functionality that
353cannot be covered by JavaScript tests.
354
355New reference tests should follow the
foolipeda32ab2017-02-16 19:21:58356[WPT reftests guidelines](http://web-platform-tests.org/writing-tests/reftests.html).
357The most important points are summarized below.
pwnall4ea2eb32016-11-29 02:47:25358
pwnall6acacd82016-12-02 01:40:15359* &#x1F6A7; The test page declares the reference page using a
360 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test
361 passes when the test image matches or does not match the reference image.
pwnall4ea2eb32016-11-29 02:47:25362* The reference page must not use the feature being tested. Otherwise, the test
363 is meaningless.
364* The reference page should be as simple as possible, and should not depend on
365 advanced features. Ideally, the reference page should render as intended even
366 on browsers with poor CSS support.
367* Reference tests should be self-describing.
368* Reference tests do _not_ include `testharness.js`.
369
pwnall6acacd82016-12-02 01:40:15370&#x1F6A7; Our testing infrastructure was designed for the
pwnall4ea2eb32016-11-29 02:47:25371[WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink
372has inherited. The consequences are summarized below.
373
374* Each reference page must be in the same directory as its associated test.
375 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`),
376 * The reference page must be named `foo-expected` (e.g.,
377 `foo-expected.html`) if the test passes when the two images match.
378 * The reference page must be named `foo-expected-mismatch` (e.g.,
379 `foo-expected-mismatch.svg`) if the test passes when the two images do
380 _not_ match.
381* Multiple references and chained references are not supported.
382
383The following example demonstrates a reference test for
384[`<ol>`'s reversed attribute](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ol).
385The example assumes that the test page is named `ol-reversed.html`.
386
387```html
388<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25389<link rel="match" href="ol-reversed-expected.html">
390
391<ol reversed>
392 <li>A</li>
393 <li>B</li>
394 <li>C</li>
395</ol>
396```
397
398The reference page, which must be named `ol-reversed-expected.html`, is below.
399
400```html
401<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25402
403<ol>
404 <li value="3">A</li>
405 <li value="2">B</li>
406 <li value="1">C</li>
407</ol>
408```
409
pwnall6acacd82016-12-02 01:40:15410*** promo
411The method for pointing out a test's reference page is still in flux, and is
412being discussed on
413[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
414***
415
pwnall4ea2eb32016-11-29 02:47:25416## Pixel Tests
417
418`testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and
419`window.testRunner.dumpDragImage()` create an image result that is associated
420with the test. The image result is compared against an image baseline, which is
421an `-expected.png` file associated with the test, and the test passes if the
422image result is identical to the baseline, according to a pixel-by-pixel
423comparison. Tests that have image results (and baselines) are called **pixel
424tests**.
425
426Pixel tests should still follow the principles laid out above. Pixel tests pose
427unique challenges to the desire to have *self-describing* and *cross-platform*
428tests. The
foolipeda32ab2017-02-16 19:21:58429[WPT rendering test guidelines](http://web-platform-tests.org/writing-tests/rendering.html)
pwnall4ea2eb32016-11-29 02:47:25430contain useful guidance. The most relevant pieces of advice are below.
431
432* Whenever possible, use a green paragraph / page / square to indicate success.
433 If that is not possible, make the test self-describing by including a textual
434 description of the desired (passing) outcome.
435* Only use the red color or the word `FAIL` to highlight errors. This does not
436 apply when testing the color red.
pwnall6acacd82016-12-02 01:40:15437* &#x1F6A7; Use the
438 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the
439 variance introduced by the platform's text rendering system. This does not
440 apply when testing text, text flow, font selection, font fallback, font
441 features, or other typographic information.
pwnall4ea2eb32016-11-29 02:47:25442
dpranked2b7d642017-01-15 04:00:24443TODO: Document how to opt out of generating a layout tree when generating
444pixel results.
445
pwnall4ea2eb32016-11-29 02:47:25446*** promo
447When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result
448will always be 800x600px, because test pages are rendered in an 800x600px
449viewport. Pixel tests that do not specifically cover scrolling should fit in an
450800x600px viewport without creating scrollbars.
451***
452
pwnall6acacd82016-12-02 01:40:15453*** promo
454The recommendation of using Ahem in pixel tests is being discussed on
455[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
456***
457
pwnall4ea2eb32016-11-29 02:47:25458The following snippet includes the Ahem font in a layout test.
459
460```html
461<style>
462body {
463 font: 10px Ahem;
464}
465</style>
466<script src="/resources/ahem.js"></script>
467```
468
469*** promo
foolip339204d2017-01-27 21:10:17470Tests outside `LayoutTests/http` and `LayoutTests/external/wpt` currently need
pwnall4ea2eb32016-11-29 02:47:25471to use a relative path to
472[/third_party/WebKit/LayoutTests/resources/ahem.js](../../third_party/WebKit/LayoutTests/resources/ahem.js)
473***
474
475### Tests that need to paint, raster, or draw a frame of intermediate output
476
477A layout test does not actually draw frames of output until the test exits.
478Tests that need to generate a painted frame can use
479`window.testRunner.displayAsyncThen`, which will run the machinery to put up a
480frame, then call the passed callback. There is also a library at
481`fast/repaint/resources/text-based-repaint.js` to help with writing paint
482invalidation and repaint tests.
483
dpranked2b7d642017-01-15 04:00:24484## Layout tree tests
pwnall4ea2eb32016-11-29 02:47:25485
dpranked2b7d642017-01-15 04:00:24486A layout tree test renders a web page and produces up to two results, which
pwnall4ea2eb32016-11-29 02:47:25487are compared against baseline files:
488
489* All tests output a textual representation of Blink's
dpranked2b7d642017-01-15 04:00:24490 [layout tree](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/render-tree-construction) (called the render tree on that page),
pwnall4ea2eb32016-11-29 02:47:25491 which is compared against an `-expected.txt` text baseline.
492* Some tests also output the image of the rendered page, which is compared
493 against an `-expected.png` image baseline, using the same method as pixel
494 tests.
495
dpranked2b7d642017-01-15 04:00:24496Whether you want a pixel test or a layout tree test depends on whether
497you care about the visual image, the details of how that image was
498constructed, or both. It is possible for multiple layout trees to produce
499the same pixel output, so it is important to make it clear in the test
500which outputs you really care about.
pwnall4ea2eb32016-11-29 02:47:25501
dpranked2b7d642017-01-15 04:00:24502TODO: Document the API used by layout tree tests to opt out of producing image
503results.
pwnall4ea2eb32016-11-29 02:47:25504
dpranked2b7d642017-01-15 04:00:24505A layout tree test passes if _all_ of its results match their baselines. Like pixel
506tests, the output of layout tree tests depends on platform-specific details,
507so layout tree tests often require per-platform baselines. Furthermore,
508since the tests obviously depend on the layout tree structure,
509that means that if we change the layout tree you have to rebaseline each
510layout tree test to see if the results are still correct and whether the test
511is still meaningful. There are actually many cases where the layout tree
512output is misstated (i.e., wrong), because people didn't want to have to update
513existing baselines and tests. This is really unfortunate and confusing.
514
515For these reasons, layout tree tests should **only** be used to cover aspects
516of the layout code that can only be tested by looking at the layout tree. Any
517combination of the other test types is preferable to a layout tree test.
518Layout tree tests are
pwnall4ea2eb32016-11-29 02:47:25519[inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so
dpranked2b7d642017-01-15 04:00:24520the repository may have some unfortunate examples of layout tree tests.
pwnall4ea2eb32016-11-29 02:47:25521
dpranked2b7d642017-01-15 04:00:24522
523The following page is an example of a layout tree test.
pwnall4ea2eb32016-11-29 02:47:25524
525```html
526<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25527<style>
528body { font: 10px Ahem; }
529span::after {
530 content: "pass";
531 color: green;
532}
533</style>
534<script src="/resources/ahem.js"></script>
535
536<p><span>Pass if a green PASS appears to the right: </span></p>
537```
538
539The most important aspects of the example are that the test page does not
540include a testing framework, and that it follows the guidelines for pixel tests.
541The test page produces the text result below.
542
543```
544layer at (0,0) size 800x600
545 LayoutView at (0,0) size 800x600
546layer at (0,0) size 800x30
547 LayoutBlockFlow {HTML} at (0,0) size 800x30
548 LayoutBlockFlow {BODY} at (8,10) size 784x10
549 LayoutBlockFlow {P} at (0,0) size 784x10
550 LayoutInline {SPAN} at (0,0) size 470x10
551 LayoutText {#text} at (0,0) size 430x10
552 text run at (0,0) width 430: "Pass if a green PASS appears to the right: "
553 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000]
554 LayoutTextFragment (anonymous) at (430,0) size 40x10
555 text run at (430,0) width 40: "pass"
556```
557
558Notice that the test result above depends on the size of the `<p>` text. The
559test page uses the Ahem font (introduced above), whose main design goal is
560consistent cross-platform rendering. Had the test used another font, its text
561baseline would have depended on the fonts installed on the testing computer, and
562on the platform's font rendering system. Please follow the pixel tests
dpranked2b7d642017-01-15 04:00:24563guidelines and write reliable layout tree tests!
pwnall4ea2eb32016-11-29 02:47:25564
dpranked2b7d642017-01-15 04:00:24565WebKit's layout tree is described in
pwnall4ea2eb32016-11-29 02:47:25566[a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/)
dpranked2b7d642017-01-15 04:00:24567on WebKit's blog. Some of the concepts there still apply to Blink's layout tree.
pwnall4ea2eb32016-11-29 02:47:25568
569## Directory Structure
570
571The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently
572lacks a strict, formal structure. The following directories have special
573meaning:
574
575* The `http/` directory hosts tests that require an HTTP server (see above).
576* The `resources/` subdirectory in every directory contains binary files, such
577 as media files, and code that is shared by multiple test files.
578
579*** note
580Some layout tests consist of a minimal HTML page that references a JavaScript
581file in `resources/`. Please do not use this pattern for new tests, as it goes
582against the minimality principle. JavaScript and CSS files should only live in
583`resources/` if they are shared by at least two test files.
584***