blob: 6399a7bbe79f94f801a9bec51a2c055c396b619a [file] [log] [blame] [view]
pwnall4ea2eb32016-11-29 02:47:251# Writing Layout Tests
2
3_Layout tests_ is a bit of a misnomer. This term is
4[a part of our WebKit heritage](https://webkit.org/blog/1452/layout-tests-theory/),
5and we use it to refer to every test that is written as a Web page (HTML, SVG,
6or XHTML) and lives in
7[third_party/WebKit/LayoutTests/](../../third_party/WebKit/LayoutTests).
8
9[TOC]
10
11## Overview
12
13Layout tests should be used to accomplish one of the following goals:
14
151. The entire surface of Blink that is exposed to the Web should be covered by
16 tests that we contribute to the
17 [Web Platform Tests Project](https://github.com/w3c/web-platform-tests)
18 (WPT). This helps us avoid regressions, and helps us identify Web Platform
19 areas where the major browsers don't have interoperable implementations.
20 Furthermore, by contributing to projects such as WPT, we share the burden of
21 writing tests with the other browser vendors, and we help all the browsers
22 get better. This is very much in line with our goal to move the Web forward.
232. When a Blink feature cannot be tested using the tools provided by WPT, and
24 cannot be easily covered by
25 [C++ unit tests](https://cs.chromium.org/chromium/src/third_party/WebKit/Source/web/tests/?q=webframetest&sq=package:chromium&type=cs),
26 the feature must be covered by layout tests, to avoid unexpected regressions.
27 These tests will use Blink-specific testing APIs that are only available in
28 [content_shell](./layout_tests_in_content_shell.md).
29
30*** promo
31If you know that Blink layout tests are upstreamed to other projects, such as
32[test262](https://github.com/tc39/test262), please update this document. Most
33importantly, our guidelines should to make it easy for our tests to be
pwnall6acacd82016-12-02 01:40:1534upstreamed. The
35[blink-dev mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/blink-dev)
36will be happy to help you harmonize our current guidelines with communal test
37repositories.
pwnall4ea2eb32016-11-29 02:47:2538***
39
40### Test Types
41
42There are four broad types of layout tests, listed in the order of preference.
43
44* *JavaScript Tests* are the layout test implementation of
45 [xUnit tests](https://en.wikipedia.org/wiki/XUnit). These tests contain
46 assertions written in JavaScript, and pass if the assertions evaluate to
47 true.
48* *Reference Tests* render a test page and a reference page, and pass if the two
49 renderings are identical, according to a pixel-by-pixel comparison. These
50 tests are less robust, harder to debug, and significantly slower than
51 JavaScript tests, and are only used when JavaScript tests are insufficient,
52 such as when testing paint code.
53* *Pixel Tests* render a test page and compare the result against a pre-rendered
dpranked2b7d642017-01-15 04:00:2454 baseline image in the repository. Pixel tests are less robust than the
55 first two types, because the rendering of a page is influenced by
pwnall4ea2eb32016-11-29 02:47:2556 many factors such as the host computer's graphics card and driver, the
57 platform's text rendering system, and various user-configurable operating
58 system settings. For this reason, it is common for a pixel test to have a
dpranked2b7d642017-01-15 04:00:2459 different reference image for each platform that Blink is tested on, and
60 the reference images are
61 [quite cumbersome to manage](./layout_test_expectations.md). You
62 should only write a pixel test if you cannot use a reference test. By default
63 a pixel test will also dump the layout tree as text output, so they are
64 similar to ...
65* *Layout tree tests*, which output a textual representation of the layout
pwnall4ea2eb32016-11-29 02:47:2566 tree, which is the key data structure in Blink's page rendering system. The
dpranked2b7d642017-01-15 04:00:2467 test passes if the output matches a baseline text file in the repository.
68 Layout tree tests are used as a last resort to test the internal quirks of
69 the implementation, and they should be avoided in favor of one of the earlier
70 options.
pwnall59aadcb2017-01-26 23:27:2171
pwnall4ea2eb32016-11-29 02:47:2572## General Principles
73
pwnall6acacd82016-12-02 01:40:1574
75Tests should be written under the assumption that they will be upstreamed
pwnall59aadcb2017-01-26 23:27:2176to the WPT project. To this end, tests should follow the
pwnall6acacd82016-12-02 01:40:1577[WPT guidelines](http://testthewebforward.org/docs/writing-tests.html).
78
pwnall6acacd82016-12-02 01:40:1579
pwnall59aadcb2017-01-26 23:27:2180There is no style guide that applies to all layout tests. However, some projects
81have adopted style guides, such as the
82[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
pwnall6acacd82016-12-02 01:40:1583
pwnall59aadcb2017-01-26 23:27:2184Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most
85important WPT guidelines and highlights some JavaScript concepts that are worth
86paying attention to when trying to infer style rules from existing tests. If
87you're unopinionated and looking for a style guide to follow, the document also
88suggests some defaults.
pwnall6acacd82016-12-02 01:40:1589
pwnall4ea2eb32016-11-29 02:47:2590## JavaScript Tests
91
92Whenever possible, the testing criteria should be expressed in JavaScript. The
93alternatives, which will be described in future sections, result in slower and
94less reliable tests.
95
96All new JavaScript tests should be written using the
97[testharness.js](https://github.com/w3c/testharness.js/) testing framework. This
98framework is used by the tests in the
99[web-platform-tests](https://github.com/w3c/web-platform-tests) repository,
100which is shared with all the other browser vendors, so `testharness.js` tests
101are more accessible to browser developers.
102
103As a shared framework, `testharness.js` enjoys high-quality documentation, such
104as [a tutorial](http://testthewebforward.org/docs/testharness-tutorial.html) and
105[API documentation](https://github.com/w3c/testharness.js/blob/master/docs/api.md).
106Layout tests should follow the recommendations of the above documents.
107Furthermore, layout tests should include relevant
108[metadata](http://testthewebforward.org/docs/css-metadata.html). The
109specification URL (in `<link rel="help">`) is almost always relevant, and is
110incredibly helpful to a developer who needs to understand the test quickly.
111
112Below is a skeleton for a JavaScript test embedded in an HTML page. Note that,
113in order to follow the minimality guideline, the test omits the tags `<html>`,
114`<head>`, and `<body>`, as they can be inferred by the HTML parser.
115
116```html
117<!doctype html>
pwnall59aadcb2017-01-26 23:27:21118<title>JavaScript: the true literal is immutable and equal to itself</title>
pwnall4ea2eb32016-11-29 02:47:25119<link rel="help" href="https://tc39.github.io/ecma262/#sec-boolean-literals">
pwnall4ea2eb32016-11-29 02:47:25120<script src="/resources/testharness.js"></script>
121<script src="/resources/testharnessreport.js"></script>
122<script>
123'use strict';
124
125// Synchronous test example.
126test(() => {
127 const value = true;
128 assert_true(value, 'true literal');
129 assert_equals(value.toString(), 'true', 'the string representation of true');
130}, 'The literal true in a synchronous test case');
131
132// Asynchronous test example.
133async_test(t => {
134 const originallyTrue = true;
135 setTimeout(t.step_func_done(() => {
136 assert_equals(originallyTrue, true);
137 }), 0);
138}, 'The literal true in a setTimeout callback');
139
140// Promise test example.
141promise_test(() => {
142 return new Promise((resolve, reject) => {
143 resolve(true);
144 }).then(value => {
145 assert_true(value);
146 });
147}, 'The literal true used to resolve a Promise');
148
149</script>
150```
151
152Some points that are not immediately obvious from the example:
153
pwnall4ea2eb32016-11-29 02:47:25154* When calling an `assert_` function that compares two values, the first
155 argument is the actual value (produced by the functionality being tested), and
156 the second argument is the expected value (known good, golden). The order
157 is important, because the testing harness relies on it to generate expressive
158 error messages that are relied upon when debugging test failures.
159* The assertion description (the string argument to `assert_` methods) conveys
160 the way the actual value was obtained.
161 * If the expected value doesn't make it clear, the assertion description
162 should explain the desired behavior.
163 * Test cases with a single assertion should omit the assertion's description
164 when it is sufficiently clear.
165* Each test case describes the circumstance that it tests, without being
166 redundant.
167 * Do not start test case descriptions with redundant terms like "Testing"
168 or "Test for".
ktyliue0bb9882017-01-10 01:47:50169 * Test files with a single test case should omit the test case description.
170 The file's `<title>` should be sufficient to describe the scenario being
171 tested.
pwnall4ea2eb32016-11-29 02:47:25172* Asynchronous tests have a few subtleties.
173 * The `async_test` wrapper calls its function with a test case argument that
174 is used to signal when the test case is done, and to connect assertion
175 failures to the correct test.
176 * `t.done()` must be called after all the test case's assertions have
177 executed.
178 * Test case assertions (actually, any callback code that can throw
179 exceptions) must be wrapped in `t.step_func()` calls, so that
180 assertion failures and exceptions can be traced back to the correct test
181 case.
182 * `t.step_func_done()` is a shortcut that combines `t.step_func()` with a
183 `t.done()` call.
184
185*** promo
186Layout tests that load from `file://` origins must currently use relative paths
187to point to
188[/resources/testharness.js](../../third_party/WebKit/LayoutTests/resources/testharness.js)
189and
190[/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js).
191This is contrary to the WPT guidelines, which call for absolute paths.
192This limitation does not apply to the tests in `LayoutTests/http`, which rely on
193an HTTP server, or to the tests in `LayoutTests/imported/wpt`, which are
194imported from the [WPT repository](https://github.com/w3c/web-platform-tests).
195***
196
197### WPT Supplemental Testing APIs
198
199Some tests simply cannot be expressed using the Web Platform APIs. For example,
200some tests that require a user to perform a gesture, such as a mouse click,
201cannot be implemented using Web APIs. The WPT project covers some of these cases
202via supplemental testing APIs.
203
pwnall59aadcb2017-01-26 23:27:21204When writing tests that rely on supplemental testing APIs, please consider the
205cost and benefits of having the tests
206[gracefully degrade to manual tests](./layout_tests_with_manual_fallback.md) in
207the absence of the testing APIs.
208
pwnall4ea2eb32016-11-29 02:47:25209*** promo
210In many cases, the user gesture is not actually necessary. For example, many
211event handling tests can use
212[synthetic events](https://developer.mozilla.org/docs/Web/Guide/Events/Creating_and_triggering_events).
213***
214
215*** note
216TODO: document wpt_automation. Manual tests might end up moving here.
217***
218
219### Relying on Blink-Specific Testing APIs
220
221Tests that cannot be expressed using the Web Platform APIs or WPT's testing APIs
222use Blink-specific testing APIs. These APIs are only available in
223[content_shell](./layout_tests_in_content_shell.md), and should only be used as
224a last resort.
225
226A downside of Blink-specific APIs is that they are not as well documented as the
227Web Platform features. Learning to use a Blink-specific feature requires finding
228other tests that use it, or reading its source code.
229
230For example, the most popular Blink-specific API is `testRunner`, which is
231implemented in
232[components/test_runner/test_runner.h](../../components/test_runner/test_runner.h)
233and
234[components/test_runner/test_runner.cpp](../../components/test_runner/test_runner.cpp).
235By skimming the `TestRunnerBindings::Install` method, we learn that the
236testRunner API is presented by the `window.testRunner` and
237`window.layoutTestsController` objects, which are synonyms. Reading the
238`TestRunnerBindings::GetObjectTemplateBuilder` method tells us what properties
239are available on the `window.testRunner` object.
240
241*** aside
242`window.testRunner` is the preferred way to access the `testRunner` APIs.
243`window.layoutTestsController` is still supported because it is used by
2443rd-party tests.
245***
246
247*** note
248`testRunner` is the most popular testing API because it is also used indirectly
249by tests that stick to Web Platform APIs. The `testharnessreport.js` file in
250`testharness.js` is specifically designated to hold glue code that connects
251`testharness.js` to the testing environment. Our implementation is in
252[third_party/WebKit/LayoutTests/resources/testharnessreport.js](../../third_party/WebKit/LayoutTests/resources/testharnessreport.js),
253and uses the `testRunner` API.
254***
255
256See the [components/test_runner/](../../components/test_runner/) directory and
257[WebKit's LayoutTests guide](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree)
258for other useful APIs. For example, `window.eventSender`
259([components/test_runner/event_sender.h](../../components/test_runner/event_sender.h)
260and
261[components/test_runner/event_sender.cpp](../../components/test_runner/event_sender.cpp))
262has methods that simulate events input such as keyboard / mouse input and
263drag-and-drop.
264
265Here is a UML diagram of how the `testRunner` bindings fit into Chromium.
266
267[![UML of testRunner bindings configuring platform implementation](https://docs.google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/svg?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
pwnall6acacd82016-12-02 01:40:15268
pwnall4ea2eb32016-11-29 02:47:25269### Text Test Baselines
270
271By default, all the test cases in a file that uses `testharness.js` are expected
272to pass. However, in some cases, we prefer to add failing test cases to the
273repository, so that we can be notified when the failure modes change (e.g., we
274want to know if a test starts crashing rather than returning incorrect output).
275In these situations, a test file will be accompanied by a baseline, which is an
276`-expected.txt` file that contains the test's expected output.
277
278The baselines are generated automatically when appropriate by
279`run-webkit-tests`, which is described [here](./layout_tests.md), and by the
280[rebaselining tools](./layout_test_expectations.md).
281
282Text baselines for `testharness.js` should be avoided, as having a text baseline
283associated with a `testharness.js` indicates the presence of a bug. For this
284reason, CLs that add text baselines must include a
285[crbug.com](https://crbug.com) link for an issue tracking the removal of the
286text expectations.
287
288* When creating tests that will be upstreamed to WPT, and Blink's current
289 behavior does not match the specification that is being tested, a text
290 baseline is necessary. Remember to create an issue tracking the expectation's
291 removal, and to link the issue in the CL description.
292* Layout tests that cannot be upstreamed to WPT should use JavaScript to
293 document Blink's current behavior, rather than using JavaScript to document
294 desired behavior and a text file to document current behavior.
295
296### The js-test.js Legacy Harness
297
298*** promo
299For historical reasons, older tests are written using the `js-test` harness.
300This harness is **deprecated**, and should not be used for new tests.
301***
302
303If you need to understand old tests, the best `js-test` documentation is its
304implementation at
305[third_party/WebKit/LayoutTests/resources/js-test.js](../../third_party/WebKit/LayoutTests/resources/js-test.js).
306
307`js-test` tests lean heavily on the Blink-specific `testRunner` testing API.
308In a nutshell, the tests call `testRunner.dumpAsText()` to signal that the page
309content should be dumped and compared against a text baseline (an
310`-expected.txt` file). As a consequence, `js-test` tests are always accompanied
311by text baselines. Asynchronous tests also use `testRunner.waitUntilDone()` and
312`testRunner.notifyDone()` to tell the testing tools when they are complete.
313
314### Tests that use an HTTP Server
315
316By default, tests are loaded as if via `file:` URLs. Some web platform features
317require tests served via HTTP or HTTPS, for example absolute paths (`src=/foo`)
318or features restricted to secure protocols.
319
320HTTP tests are those under `LayoutTests/http/tests` (or virtual variants). Use a
321locally running HTTP server (Apache) to run them. Tests are served off of ports
3228000 and 8080 for HTTP, and 8443 for HTTPS. If you run the tests using
323`run-webkit-tests`, the server will be started automatically. To run the server
324manually to reproduce or debug a failure:
325
326```bash
327cd src/third_party/WebKit/Tools/Scripts
328run-blink-httpd start
329```
330
331The layout tests will be served from `http://127.0.0.1:8000`. For example, to
332run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
333navigate to
334`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
335tests will behave differently if you go to 127.0.0.1 instead of localhost, so
336use 127.0.0.1.
337
338To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill`
339or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS.
340
341The test server sets up an alias to the `LayoutTests/resources` directory. In
342HTTP tests, you can access the testing framework at e.g.
pwnalle7819482016-12-17 00:58:40343`src="/resources/testharness.js"`.
pwnall4ea2eb32016-11-29 02:47:25344
345TODO: Document [wptserve](http://wptserve.readthedocs.io/) when we are in a
346position to use it to run layout tests.
347
348## Reference Tests (Reftests)
349
350Reference tests, also known as reftests, perform a pixel-by-pixel comparison
351between the rendered image of a test page and the rendered image of a reference
352page. Most reference tests pass if the two images match, but there are cases
353where it is useful to have a test pass when the two images do _not_ match.
354
355Reference tests are more difficult to debug than JavaScript tests, and tend to
356be slower as well. Therefore, they should only be used for functionality that
357cannot be covered by JavaScript tests.
358
359New reference tests should follow the
360[WPT reftests guidelines](http://testthewebforward.org/docs/reftests.html). The
361most important points are summarized below.
362
pwnall6acacd82016-12-02 01:40:15363* &#x1F6A7; The test page declares the reference page using a
364 `<link rel="match">` or `<link rel="mismatch">`, depending on whether the test
365 passes when the test image matches or does not match the reference image.
pwnall4ea2eb32016-11-29 02:47:25366* The reference page must not use the feature being tested. Otherwise, the test
367 is meaningless.
368* The reference page should be as simple as possible, and should not depend on
369 advanced features. Ideally, the reference page should render as intended even
370 on browsers with poor CSS support.
371* Reference tests should be self-describing.
372* Reference tests do _not_ include `testharness.js`.
373
pwnall6acacd82016-12-02 01:40:15374&#x1F6A7; Our testing infrastructure was designed for the
pwnall4ea2eb32016-11-29 02:47:25375[WebKit reftests](https://trac.webkit.org/wiki/Writing%20Reftests) that Blink
376has inherited. The consequences are summarized below.
377
378* Each reference page must be in the same directory as its associated test.
379 Given a test page named `foo` (e.g. `foo.html` or `foo.svg`),
380 * The reference page must be named `foo-expected` (e.g.,
381 `foo-expected.html`) if the test passes when the two images match.
382 * The reference page must be named `foo-expected-mismatch` (e.g.,
383 `foo-expected-mismatch.svg`) if the test passes when the two images do
384 _not_ match.
385* Multiple references and chained references are not supported.
386
387The following example demonstrates a reference test for
388[`<ol>`'s reversed attribute](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ol).
389The example assumes that the test page is named `ol-reversed.html`.
390
391```html
392<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25393<link rel="match" href="ol-reversed-expected.html">
394
395<ol reversed>
396 <li>A</li>
397 <li>B</li>
398 <li>C</li>
399</ol>
400```
401
402The reference page, which must be named `ol-reversed-expected.html`, is below.
403
404```html
405<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25406
407<ol>
408 <li value="3">A</li>
409 <li value="2">B</li>
410 <li value="1">C</li>
411</ol>
412```
413
pwnall6acacd82016-12-02 01:40:15414*** promo
415The method for pointing out a test's reference page is still in flux, and is
416being discussed on
417[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
418***
419
pwnall4ea2eb32016-11-29 02:47:25420## Pixel Tests
421
422`testRunner` APIs such as `window.testRunner.dumpAsTextWithPixelResults()` and
423`window.testRunner.dumpDragImage()` create an image result that is associated
424with the test. The image result is compared against an image baseline, which is
425an `-expected.png` file associated with the test, and the test passes if the
426image result is identical to the baseline, according to a pixel-by-pixel
427comparison. Tests that have image results (and baselines) are called **pixel
428tests**.
429
430Pixel tests should still follow the principles laid out above. Pixel tests pose
431unique challenges to the desire to have *self-describing* and *cross-platform*
432tests. The
433[WPT test style guidelines](http://testthewebforward.org/docs/test-style-guidelines.html)
434contain useful guidance. The most relevant pieces of advice are below.
435
436* Whenever possible, use a green paragraph / page / square to indicate success.
437 If that is not possible, make the test self-describing by including a textual
438 description of the desired (passing) outcome.
439* Only use the red color or the word `FAIL` to highlight errors. This does not
440 apply when testing the color red.
pwnall6acacd82016-12-02 01:40:15441* &#x1F6A7; Use the
442 [Ahem font](https://www.w3.org/Style/CSS/Test/Fonts/Ahem/README) to reduce the
443 variance introduced by the platform's text rendering system. This does not
444 apply when testing text, text flow, font selection, font fallback, font
445 features, or other typographic information.
pwnall4ea2eb32016-11-29 02:47:25446
dpranked2b7d642017-01-15 04:00:24447TODO: Document how to opt out of generating a layout tree when generating
448pixel results.
449
pwnall4ea2eb32016-11-29 02:47:25450*** promo
451When using `window.testRunner.dumpAsTextWithPixelResults()`, the image result
452will always be 800x600px, because test pages are rendered in an 800x600px
453viewport. Pixel tests that do not specifically cover scrolling should fit in an
454800x600px viewport without creating scrollbars.
455***
456
pwnall6acacd82016-12-02 01:40:15457*** promo
458The recommendation of using Ahem in pixel tests is being discussed on
459[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
460***
461
pwnall4ea2eb32016-11-29 02:47:25462The following snippet includes the Ahem font in a layout test.
463
464```html
465<style>
466body {
467 font: 10px Ahem;
468}
469</style>
470<script src="/resources/ahem.js"></script>
471```
472
473*** promo
474Tests outside `LayoutTests/http` and `LayoutTests/imported/wpt` currently need
475to use a relative path to
476[/third_party/WebKit/LayoutTests/resources/ahem.js](../../third_party/WebKit/LayoutTests/resources/ahem.js)
477***
478
479### Tests that need to paint, raster, or draw a frame of intermediate output
480
481A layout test does not actually draw frames of output until the test exits.
482Tests that need to generate a painted frame can use
483`window.testRunner.displayAsyncThen`, which will run the machinery to put up a
484frame, then call the passed callback. There is also a library at
485`fast/repaint/resources/text-based-repaint.js` to help with writing paint
486invalidation and repaint tests.
487
dpranked2b7d642017-01-15 04:00:24488## Layout tree tests
pwnall4ea2eb32016-11-29 02:47:25489
dpranked2b7d642017-01-15 04:00:24490A layout tree test renders a web page and produces up to two results, which
pwnall4ea2eb32016-11-29 02:47:25491are compared against baseline files:
492
493* All tests output a textual representation of Blink's
dpranked2b7d642017-01-15 04:00:24494 [layout tree](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/render-tree-construction) (called the render tree on that page),
pwnall4ea2eb32016-11-29 02:47:25495 which is compared against an `-expected.txt` text baseline.
496* Some tests also output the image of the rendered page, which is compared
497 against an `-expected.png` image baseline, using the same method as pixel
498 tests.
499
dpranked2b7d642017-01-15 04:00:24500Whether you want a pixel test or a layout tree test depends on whether
501you care about the visual image, the details of how that image was
502constructed, or both. It is possible for multiple layout trees to produce
503the same pixel output, so it is important to make it clear in the test
504which outputs you really care about.
pwnall4ea2eb32016-11-29 02:47:25505
dpranked2b7d642017-01-15 04:00:24506TODO: Document the API used by layout tree tests to opt out of producing image
507results.
pwnall4ea2eb32016-11-29 02:47:25508
dpranked2b7d642017-01-15 04:00:24509A layout tree test passes if _all_ of its results match their baselines. Like pixel
510tests, the output of layout tree tests depends on platform-specific details,
511so layout tree tests often require per-platform baselines. Furthermore,
512since the tests obviously depend on the layout tree structure,
513that means that if we change the layout tree you have to rebaseline each
514layout tree test to see if the results are still correct and whether the test
515is still meaningful. There are actually many cases where the layout tree
516output is misstated (i.e., wrong), because people didn't want to have to update
517existing baselines and tests. This is really unfortunate and confusing.
518
519For these reasons, layout tree tests should **only** be used to cover aspects
520of the layout code that can only be tested by looking at the layout tree. Any
521combination of the other test types is preferable to a layout tree test.
522Layout tree tests are
pwnall4ea2eb32016-11-29 02:47:25523[inherited from WebKit](https://webkit.org/blog/1456/layout-tests-practice/), so
dpranked2b7d642017-01-15 04:00:24524the repository may have some unfortunate examples of layout tree tests.
pwnall4ea2eb32016-11-29 02:47:25525
dpranked2b7d642017-01-15 04:00:24526
527The following page is an example of a layout tree test.
pwnall4ea2eb32016-11-29 02:47:25528
529```html
530<!doctype html>
pwnall4ea2eb32016-11-29 02:47:25531<style>
532body { font: 10px Ahem; }
533span::after {
534 content: "pass";
535 color: green;
536}
537</style>
538<script src="/resources/ahem.js"></script>
539
540<p><span>Pass if a green PASS appears to the right: </span></p>
541```
542
543The most important aspects of the example are that the test page does not
544include a testing framework, and that it follows the guidelines for pixel tests.
545The test page produces the text result below.
546
547```
548layer at (0,0) size 800x600
549 LayoutView at (0,0) size 800x600
550layer at (0,0) size 800x30
551 LayoutBlockFlow {HTML} at (0,0) size 800x30
552 LayoutBlockFlow {BODY} at (8,10) size 784x10
553 LayoutBlockFlow {P} at (0,0) size 784x10
554 LayoutInline {SPAN} at (0,0) size 470x10
555 LayoutText {#text} at (0,0) size 430x10
556 text run at (0,0) width 430: "Pass if a green PASS appears to the right: "
557 LayoutInline {<pseudo:after>} at (0,0) size 40x10 [color=#008000]
558 LayoutTextFragment (anonymous) at (430,0) size 40x10
559 text run at (430,0) width 40: "pass"
560```
561
562Notice that the test result above depends on the size of the `<p>` text. The
563test page uses the Ahem font (introduced above), whose main design goal is
564consistent cross-platform rendering. Had the test used another font, its text
565baseline would have depended on the fonts installed on the testing computer, and
566on the platform's font rendering system. Please follow the pixel tests
dpranked2b7d642017-01-15 04:00:24567guidelines and write reliable layout tree tests!
pwnall4ea2eb32016-11-29 02:47:25568
dpranked2b7d642017-01-15 04:00:24569WebKit's layout tree is described in
pwnall4ea2eb32016-11-29 02:47:25570[a series of posts](https://webkit.org/blog/114/webcore-rendering-i-the-basics/)
dpranked2b7d642017-01-15 04:00:24571on WebKit's blog. Some of the concepts there still apply to Blink's layout tree.
pwnall4ea2eb32016-11-29 02:47:25572
573## Directory Structure
574
575The [LayoutTests directory](../../third_party/WebKit/LayoutTests) currently
576lacks a strict, formal structure. The following directories have special
577meaning:
578
579* The `http/` directory hosts tests that require an HTTP server (see above).
580* The `resources/` subdirectory in every directory contains binary files, such
581 as media files, and code that is shared by multiple test files.
582
583*** note
584Some layout tests consist of a minimal HTML page that references a JavaScript
585file in `resources/`. Please do not use this pattern for new tests, as it goes
586against the minimality principle. JavaScript and CSS files should only live in
587`resources/` if they are shared by at least two test files.
588***