Separate layout test docs stating facts from guidelines.
BUG=
Review-Url: https://codereview.chromium.org/2636213003
Cr-Commit-Position: refs/heads/master@{#446491}
diff --git a/docs/testing/layout_tests_tips.md b/docs/testing/layout_tests_tips.md
new file mode 100644
index 0000000..8a2ee128
--- /dev/null
+++ b/docs/testing/layout_tests_tips.md
@@ -0,0 +1,301 @@
+# Layout Tests Tips
+
+The recommendations here are intended to help you write new tests that go
+through code review with a minimal number of round trips, remain useful as Blink
+evolves, and serve as an asset (rather than a liability) for the team.
+
+While reading existing layout tests, please keep in mind that they represent
+snapshots taken over many years of an ever-evolving collective opinion of what
+good Web pages and solid tests should look like. Thus, it should not come as a
+surprise that most existing layout tests are not consistent with these
+recommendations, and are not even consistent with each other.
+
+*** note
+This document intentionally uses _should_ a lot more than _must_, as defined in
+[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a
+careful act of balancing many concerns, and this humble document cannot possibly
+capture the context that rests in the head of an experienced Blink engineer.
+***
+
+## General Principles
+
+This section contains guidelines adopted from
+[Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/docs/test-format-guidelines.html)
+and
+[WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree),
+with Blink-specific flavoring.
+
+### Concise
+
+Tests should be **concise**, without compromising on the principles below. Every
+element and piece of code on the page should be necessary and relevant to what
+is being tested. For example, don't build a fully functional signup form if you
+only need a text field or a button.
+
+Content needed to satisfy the principles below is considered necessary. For
+example, it is acceptable and desirable to add elements that make the test
+self-describing (see below), and to add code that makes the test more reliable
+(see below).
+
+Content that makes test failures easier to debug is considered necessary (to
+maintaining a good development speed), and is both acceptable and desirable.
+
+*** promo
+Conciseness is particularly important for reference tests and pixel tests, as
+the test pages are rendered in an 800x600px viewport. Having content outside the
+viewport is undesirable because the outside content does not get compared, and
+because the resulting scrollbars are platform-specific UI widgets, making the
+test results less reliable.
+***
+
+### Fast
+
+Tests should be as **fast** as possible, without compromising on the principles
+below. Blink has several thousand layout tests that are run in parallel, and
+avoiding unnecessary delays is crucial to keeping our Commit Queue in good
+shape.
+
+Avoid
+[window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimers/setTimeout),
+as it wastes time on the testing infrastructure. Instead, use specific event
+handlers, such as
+[window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/onload),
+to decide when to advance to the next step in a test.
+
+### Reliable
+
+Tests should be **reliable** and yield consistent results for a given
+implementation. Flaky tests slow down your fellow developers' debugging efforts
+and the Commit Queue.
+
+`window.setTimeout` is again a primary offender here. Asides from wasting time
+on a fast system, tests that rely on fixed timeouts can fail when on systems
+that are slower than expected.
+
+When adding or significantly modifying a layout test, use the command below to
+assess its flakiness. While not foolproof, this approach gives you some
+confidence, and giving up CPU cycles for mental energy is a pretty good trade.
+
+```bash
+third_party/WebKit/Tools/Scripts/run-webkit-tests path/to/test.html --repeat-each=100
+```
+
+The
+[PSA on writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
+also has good guidelines for writing reliable tests.
+
+### Self-Describing
+
+Tests should be **self-describing**, so that a project member can recognize
+whether a test passes or fails without having to read the specification of the
+feature being tested.
+
+`testharness.js` makes a test self-describing when used correctly. Other types
+of tests, such as reference tests and
+[tests with manual fallback](./layout_tests_with_manual_fallback.md),
+[must be carefully designed](http://testthewebforward.org/docs/test-style-guidelines.html)
+to be self-describing.
+
+### Minimal
+
+Tests should require a **minimal** amount of cognitive effort to read and
+maintain.
+
+Avoid depending on edge case behavior of features that aren't explicitly covered
+by the test. For example, except where testing parsing, tests should contain
+valid markup (no parsing errors).
+
+Tests should provide as much relevant information as possible when failing.
+`testharness.js` tests should prefer
+[rich assert_ functions](https://github.com/w3c/testharness.js/blob/master/docs/api.md#list-of-assertions)
+to combining `assert_true()` with a boolean operator. Using appropriate
+`assert_` functions results in better diagnostic output when the assertion
+fails.
+
+### Cross-Platform
+
+Tests should be as **cross-platform** as reasonably possible. Avoid assumptions
+about device type, screen resolution, etc. Unavoidable assumptions should be
+documented.
+
+When possible, tests should only use Web platform features, as specified
+in the relevant standards. When the Web platform's APIs are insufficient,
+tests should prefer to use WPT extended testing APIs, such as
+`wpt_automation`, over Blink-specific testing APIs.
+
+Test pages should use the HTML5 doctype (`<!doctype html>`) unless they
+specifically cover
+[quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards_Mode)
+behavior.
+
+Tests should avoid using features that haven't been shipped by the
+actively-developed major rendering engines (Blink, WebKit, Gecko, Edge). When
+unsure, check [caniuse.com](http://caniuse.com/). By necessity, this
+recommendation does not apply to the feature targeted by the test.
+
+*** note
+It may be tempting have a test for a bleeding-edge feature X depend on feature
+Y, which has only shipped in beta / development versions of various browsers.
+The reasoning would be that all browsers that implement X will have implemented
+Y. Please keep in mind that Chrome has un-shipped features that made it to the
+Beta channel in the past.
+***
+
+*** aside
+[ES2015](http://benmccormick.org/2015/09/14/es5-es6-es2016-es-next-whats-going-on-with-javascript-versioning/)
+is shipped by all major browsers under active development (except for modules),
+so using ES2015 features is acceptable.
+
+At the time of this writing, ES2016 is not fully shipped in all major browsers.
+***
+
+### Self-Contained
+
+Tests must be **self-contained** and not depend on external network resources.
+
+Unless used by multiple test files, CSS and JavaScript should be inlined using
+`<style>` and `<script>` tags. Content shared by multiple tests should be
+placed in a `resources/` directory near the tests that share it. See below for
+using multiple origins in a test.
+
+### File Names
+
+Test **file names** should describe what is being tested.
+
+File names should use `snake-case`, but preserve the case of any embedded API
+names. For example, prefer `document-createElement.html` to
+`document-create-element.html`.
+
+### Character Encoding
+
+Tests should use the UTF-8 **character encoding**, which should be declared by
+`<meta charset=utf-8>`. A `<meta>` tag is not required (but is acceptable) for
+tests that only contain ASCII characters. This guideline does not apply when
+specifically testing encodings.
+
+The `<meta>` tag must be the first child of the document's `<head>` element. In
+documents that do not have an explicit `<head>`, the `<meta>` tag must follow
+the doctype.
+
+## Coding Style
+
+No coding style is enforced for layout tests. This section highlights coding
+style aspects that are not consistent across our layout tests, and suggests some
+defaults for unopinionated developers. When writing layout tests for a new part
+of the codebase, you can minimize review latency by taking a look at existing
+tests, and pay particular attention to these issues. Also beware of per-project
+style guides, such as the
+[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
+
+### Baseline
+
+[Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.html)
+and
+[Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide.xml)
+are a reasonable baseline for coding style defaults, with the caveat that layout
+tests do not use Google Closure or JSDoc.
+
+### == vs ===
+
+JavaScript's
+[== operator](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comparison_Operators#Equality_())
+performs some
+[type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abstract-equality-comparison).
+on its arguments, which might be surprising to readers whose experience centers
+around C++ or Java. The
+[=== operator](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comparison_Operators#Identity_strict_equality_())
+is much more similar to `==` in C++.
+
+Using `===` everywhere is an easy default that saves you, your reviewer, and any
+colleague that might have to debug test failures, from having to reason about
+[special cases for ==](http://dorey.github.io/JavaScript-Equality-Table/). At
+the same time, some developers consider `===` to add unnecessary noise when `==`
+would suffice. While `===` should be universally accepted, be flexible if your
+reviewer expresses a strong preference for `==`.
+
+### Let and Const vs Var
+
+JavaScript variable declarations introduced by
+[var](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/var)
+are hoisted to the beginning of their containing function, which may be
+surprising to C++ and Java developers. By contrast,
+[const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/const)
+and
+[let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/let)
+declarations are block-scoped, just like in C++ and Java, and have the added
+benefit of expressing mutability intent.
+
+For the reasons above, a reasonable default is to prefer `const` and `let` over
+`var`, with the same caveat as above.
+
+### Strict Mode
+
+JavaScript's
+[strict mode](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Strict_mode),
+activated by adding `'use strict';` to the very top of a script, helps catch
+some errors, such as mistyping a variable name, forgetting to declare a
+variable, or attempting to change a read-only property.
+
+Given that strict mode gives some of the benefits of using a compiler, adding it
+to every test is a good default. This does not apply when specifically testing
+sloppy mode behavior.
+
+Some developers argue that adding the `'use strict';` boilerplate can be
+difficult to remember, weighs down smaller tests, and in many cases running a
+test case is sufficient to discover any mistyped variable names.
+
+### Promises
+
+[Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
+are a mechanism for structuring asynchronous code. When used correctly, Promises
+avoid some of the
+[issues of callbacks](http://colintoh.com/blog/staying-sane-with-asynchronous-programming-promises-and-generators).
+For these reasons, a good default is to prefer promises over other asynchronous
+code structures.
+
+When using promises, be aware of the
+[execution order subtleties](https://jakearchibald.com/2015/tasks-microtasks-queues-and-schedules/)
+associated with them. Here is a quick summary.
+
+* The function passed to `Promise.new` is executed synchronously, so it finishes
+ before the Promise is created and returned.
+* The functions passed to `then` and `catch` are executed in
+ _separate microtasks_, so they will be executed after the code that resolved
+ or rejected the promise finishes, but before any other event handler.
+
+### Classes
+
+[Classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Classes)
+are syntactic sugar for JavaScript's
+[prototypal inheritance](https://developer.mozilla.org/docs/Web/JavaScript/Inheritance_and_the_prototype_chain).
+Compared to manipulating prototypes directly, classes offer a syntax that is
+more familiar to developers coming from other programming languages.
+
+A good default is to prefer classes over other OOP constructs, as they will make
+the code easier to read for many of your fellow Chrome developers. At the same
+time, most layout tests are simple enough that OOP is not justified.
+
+### Character Encoding
+
+When HTML pages do not explicitly declare a character encoding, browsers
+determine the encoding using an
+[encoding sniffing algorithm](https://html.spec.whatwg.org/multipage/syntax.html#determining-the-character-encoding)
+that will surprise most modern Web developers. Highlights include a default
+encoding that depends on the user's locale, and non-standardized
+browser-specific heuristics.
+
+The easiest way to not have to think about any of this is to add
+`<meta charset="utf-8">` to all your tests. This is easier to remember if you
+use a template for your layout tests, rather than writing them from scratch.
+
+## Tests with Manual Feedback
+
+Tests that rely on the testing APIs exposed by WPT or Blink will not work when
+loaded in a standard browser environment. When writing such tests, default to
+having the tests gracefully degrade to manual tests in the absence of the
+testing APIs.
+
+The
+[document on layout tests with manual feedback](./layout_tests_with_manual_fallback.md)
+describes the approach in detail and highlights the trade-off between added test
+weight and ease of debugging.
diff --git a/docs/testing/layout_tests_with_manual_fallback.md b/docs/testing/layout_tests_with_manual_fallback.md
new file mode 100644
index 0000000..1a51d5a0
--- /dev/null
+++ b/docs/testing/layout_tests_with_manual_fallback.md
@@ -0,0 +1,119 @@
+# Layout Tests with Manual Fallback
+
+Some Blink features cannot be automatically tested using the Web Platform. Prime
+examples are the APIs that require
+[user activation](https://html.spec.whatwg.org/multipage/interaction.html#triggered-by-user-activation)
+(also known as _a user gesture_), such as [Full Screen](https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API).
+Automated tests for these Blink features must rely on special APIs, which are
+only exposed in testing environments, and are therefore not available in a
+normal browser session.
+
+A popular pattern used in these tests is to rely on the user to perform some
+manual steps in order to run the test case in a normal browser session. These
+tests are effectively
+[manual tests](http://testthewebforward.org/docs/manual-test.html), with
+additional JavaScript code that automatically performs the desired manual steps,
+when loaded in an environment that exposes the needed testing APIs.
+
+## Motivation
+
+Layout tests that degrade to manual tests in the absence of testing APIs have
+the following benefits.
+
+* The manual test component can be debugged in a normal browser session, using
+ the rich [developer tools](https://developer.chrome.com/devtools). Tests
+ without a manual fallback can only be debugged in the test runner.
+* The manual tests can run in other browsers, making it easy to check whether
+ our behavior matches other browsers.
+* The layout tests can form the basis for manual tests that are contributed to
+ the [Web Platform Tests Project](https://github.com/w3c/web-platform-tests).
+
+Therefore, the desirability of adding a manual fallback to a test heavily
+depends on whether the feature under test is a Web Platform feature or a
+Blink-only feature, and on the developer's working style. The benefits above
+should be weighed against the added design effort needed to build a manual test,
+and the size and complexity introduced by the manual fallback.
+
+## Development Tips
+
+A natural workflow for writing a layout test that gracefully degrades to a
+manual test is to first develop the manual test in a browser, and then add code
+that feature-checks for testing APIs, and uses them to automate the test's
+manual steps.
+
+Manual tests should minimize the chance of user error. This implies keeping the
+manual steps to a minimum, and having simple and clear instructions that
+describe all the configuration changes and user gestures that match the effect
+of the Blink-specific APIs used by the test.
+
+## Example
+
+Below is an example of a fairly minimal test that uses a Blink-Specific API
+(`window.eventSender`), and gracefully degrades to a manual test.
+
+```html
+<!doctype html>
+<meta charset="utf-8">
+<title>DOM: Event.isTrusted for UI events</title>
+<link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
+<link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
+<meta name="assert"
+ content="Event.isTrusted is true for events generated by user interaction">
+<script src="../../resources/testharness.js"></script>
+<script src="../../resources/testharnessreport.js"></script>
+
+<p>Please click on the button below.</p>
+<button>Click Me!</button>
+
+<script>
+'use strict';
+
+setup({ explicit_timeout: true });
+
+promise_test(() => {
+ const button = document.querySelector('button');
+ return new Promise((resolve, reject) => {
+ const button = document.querySelector('button');
+ button.addEventListener('click', (event) => {
+ resolve(event);
+ });
+
+ if (window.eventSender) {
+ eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
+ eventSender.mouseDown();
+ eventSender.mouseUp();
+ }
+ }).then((clickEvent) => {
+ assert_true(clickEvent.isTrusted);
+ });
+
+}, 'Click generated by user interaction');
+
+</script>
+```
+
+The test exhibits the following desirable features:
+
+* It has a second specification URL (`<link rel="help">`), because the paragraph
+ that documents the tested feature (referenced by the primary URL) is not very
+ informative on its own.
+* It links to the
+ [WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F),
+ rather than to a frozen version of the specification.
+* It contains clear instructions for manually triggering the test conditions.
+ The test starts with a paragraph (`<p>`) that tells the tester exactly what to
+ do, and the `<button>` that needs to be clicked is clearly labeled.
+* It disables the timeout mechanism built into `testharness.js` by calling
+ `setup({ explicit_timeout: true });`
+* It checks for the presence of the Blink-specific testing APIs
+ (`window.eventSender`) before invoking them. The test does not automatically
+ fail when the APIs are not present.
+* It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
+ to separate the test setup from the assertions. This is particularly helpful
+ for manual tests that depend on a sequence of events to occur, as Promises
+ offer a composable way to express waiting for asynchronous events that avoids
+ [callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).
+
+Notice that the test is pretty heavy compared to a minimal JavaScript test that
+does not rely on testing APIs. Only use testing APIs when the desired testing
+conditions cannot be set up using Web Platform APIs.
diff --git a/docs/testing/writing_layout_tests.md b/docs/testing/writing_layout_tests.md
index 37e5e2a..6399a7b 100644
--- a/docs/testing/writing_layout_tests.md
+++ b/docs/testing/writing_layout_tests.md
@@ -68,246 +68,24 @@
Layout tree tests are used as a last resort to test the internal quirks of
the implementation, and they should be avoided in favor of one of the earlier
options.
+
## General Principles
-The principles below are adapted from
-[Test the Web Forward's Test Format Guidelines](http://testthewebforward.org/docs/test-format-guidelines.html)
-and
-[WebKit's Wiki page on Writing good test cases](https://trac.webkit.org/wiki/Writing%20Layout%20Tests%20for%20DumpRenderTree).
-
-*** note
-This document intentionally uses _should_ a lot more than _must_, as defined in
-[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). Writing layout tests is a
-careful act of balancing many concerns, and this humble document cannot possibly
-capture the context that rests in the head of an experienced Blink engineer.
-***
-
-### Concise
-
-Tests should be **concise**, without compromising on the principles below. Every
-element and piece of code on the page should be necessary and relevant to what
-is being tested. For example, don't build a fully functional signup form if you
-only need a text field or a button.
-
-Content needed to satisfy the principles below is considered necessary. For
-example, it is acceptable and desirable to add elements that make the test
-self-describing (see below), and to add code that makes the test more reliable
-(see below).
-
-Content that makes test failures easier to debug is considered necessary (to
-maintaining a good development speed), and is both acceptable and desirable.
-
-*** promo
-Conciseness is particularly important for reference tests and pixel tests, as
-the test pages are rendered in an 800x600px viewport. Having content outside the
-viewport is undesirable because the outside content does not get compared, and
-because the resulting scrollbars are platform-specific UI widgets, making the
-test results less reliable.
-***
-
-### Fast
-
-Tests should be as **fast** as possible, without compromising on the principles
-below. Blink has several thousand layout tests that are run in parallel, and
-avoiding unnecessary delays is crucial to keeping our Commit Queue in good
-shape.
-
-Avoid
-[window.setTimeout](https://developer.mozilla.org/en-US/docs/Web/API/WindowTimers/setTimeout),
-as it wastes time on the testing infrastructure. Instead, use specific event
-handlers, such as
-[window.onload](https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/onload),
-to decide when to advance to the next step in a test.
-
-### Reliable
-
-Tests should be **reliable** and yield consistent results for a given
-implementation. Flaky tests slow down your fellow developers' debugging efforts
-and the Commit Queue.
-
-`window.setTimeout` is again a primary offender here. Asides from wasting time
-on a fast system, tests that rely on fixed timeouts can fail when on systems
-that are slower than expected.
-
-Follow the guidelines in this
-[PSA on writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit).
-
-### Self-Describing
-
-Tests should be **self-describing**, so that a project member can recognize
-whether a test passes or fails without having to read the specification of the
-feature being tested. `testharness.js` makes a test self-describing when used
-correctly, but tests that degrade to manual tests
-[must be carefully designed](http://testthewebforward.org/docs/test-style-guidelines.html)
-to be self-describing.
-
-### Minimal
-
-Tests should require a **minimal** amount of cognitive effort to read and
-maintain.
-
-Avoid depending on edge case behavior of features that aren't explicitly covered
-by the test. For example, except where testing parsing, tests should contain
-valid markup (no parsing errors).
-
-Tests should provide as much relevant information as possible when failing.
-`testharness.js` tests should prefer
-[rich assert_ functions](https://github.com/w3c/testharness.js/blob/master/docs/api.md#list-of-assertions)
-to combining `assert_true()` with a boolean operator. Using appropriate
-`assert_` functions results in better diagnostic output when the assertion
-fails.
-
-🚧 Prefer JavaScript's
-[===](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comparison_Operators#Identity_strict_equality_())
-operator to
-[==](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Operators/Comparison_Operators#Equality_())
-so that readers don't have to reason about
-[type conversion](http://www.ecma-international.org/ecma-262/6.0/#sec-abstract-equality-comparison).
-
-*** promo
-The === vs == recommendation is still being discussed on
-[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
-However, please keep in mind that a fellow developer who has to debug a failing
-test may have to consider the
-[special cases for ==](http://dorey.github.io/JavaScript-Equality-Table/) when
-the types of the two values being compared aren't immediately obvious.
-***
-
-### Cross-Platform
-
-Tests should be as **cross-platform** as reasonably possible. Avoid assumptions
-about device type, screen resolution, etc. Unavoidable assumptions should be
-documented.
-
-When possible, tests should only use Web platform features, as specified
-in the relevant standards. When the Web platform's APIs are insufficient,
-tests should prefer to use WPT extended testing APIs, such as
-`wpt_automation`, over Blink-specific testing APIs.
-
-🚧 Tests that use testing APIs should feature-test for the presence of
-those APIs, and gracefully degrade to manual tests (see below) when the testing
-APIs are not available.
-
-Test pages should use the HTML5 doctype (`<!doctype html>`) unless they
-specifically cover
-[quirks mode](https://developer.mozilla.org/docs/Quirks_Mode_and_Standards_Mode)
-behavior.
Tests should be written under the assumption that they will be upstreamed
-to the WPT project. For example, tests should follow the
+to the WPT project. To this end, tests should follow the
[WPT guidelines](http://testthewebforward.org/docs/writing-tests.html).
-Tests should avoid using features that haven't been shipped by the
-actively-developed major rendering engines (Blink, WebKit, Gecko, Edge). When
-unsure, check [caniuse.com](http://caniuse.com/). By necessity, this
-recommendation does not apply to the feature targeted by the test.
-*** note
-It may be tempting have a test for a bleeding-edge feature X depend on feature
-Y, which has only shipped in beta / development versions of various browsers.
-The reasoning would be that all browsers that implement X will have implemented
-Y. Please keep in mind that Chrome has un-shipped features that made it to the
-Beta channel in the past.
-***
+There is no style guide that applies to all layout tests. However, some projects
+have adopted style guides, such as the
+[ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
-Tests that use Blink-specific testing APIs should feature-test for the
-presence of the testing APIs and degrade to
-[manual tests](http://testthewebforward.org/docs/manual-test.html) when
-the testing APIs are not present.
-
-*** promo
-The recommendation to degrade to manual tests is still being discussed on
-[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
-However, please keep in mind that a manual test can be debugged in the browser,
-whereas a test that does not degrade gracefully can only be debugged in the test
-runner. Fellow project members and future you will thank you for having your
-test work as a manual test.
-***
-
-### Self-Contained
-
-Tests must be **self-contained** and not depend on external network resources.
-
-Unless used by multiple test files, CSS and JavaScript should be inlined using
-`<style>` and `<script>` tags. Content shared by multiple tests should be
-placed in a `resources/` directory near the tests that share it. See below for
-using multiple origins in a test.
-
-### File Names
-
-Test **file names** should describe what is being tested.
-
-File names should use `snake-case`, but preserve the case of any embedded API
-names. For example, prefer `document-createElement.html` to
-`document-create-element.html`.
-
-### Modern Features
-
-Tests should prefer **modern features** in JavaScript and in the Web Platform,
-provided that they meet the recommendations above for cross-platform tests.
-
-Tests should use
-[strict mode](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode)
-for all JavaScript, except when specifically testing sloppy mode behavior.
-Strict mode flags deprecated features and helps catch some errors, such as
-forgetting to declare variables.
-
-🚧 JavaScript code should prefer
-[const](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/const)
-and
-[let](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/let)
-over `var`,
-[classes](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Classes)
-over other OOP constructs, and
-[Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
-over other mechanisms for structuring asynchronous code.
-
-*** promo
-The recommendation to prefer `const` and `let` over `var` is currently being
-discussed on
-[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
-***
-
-### Character Encoding
-
-🚧 Tests should use the UTF-8 **character encoding**, which should be
-declared by `<meta charset=utf-8>`. This does not apply when specifically
-testing encodings.
-
-The `<meta>` tag must be the first child of the document's `<head>` element. In
-documents that do not have an explicit `<head>`, the `<meta>` tag must follow
-the doctype.
-
-When HTML pages do not explicitly declare a character encoding, browsers
-determine the encoding using an
-[encoding sniffing algorithm](https://html.spec.whatwg.org/multipage/syntax.html#determining-the-character-encoding)
-that will surprise most modern Web developers. Highlights include a default
-encoding that depends on the user's locale, and non-standardized
-browser-specific heuristics.
-
-*** promo
-The WPT guidelines state that test files that only contain ASCII characters may
-omit the `<meta>` tag. This exception is currently discussed on
-[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
-If taking that route, please keep in mind that Firefox currently issues a
-[development tools](https://developer.mozilla.org/en-US/docs/Tools) warning for
-pages without a declared encoding.
-***
-
-### Coding Style
-
-Tests should aim to have a **coding style** that is consistent with
-[Google's JavaScript Style Guide](https://google.github.io/styleguide/jsguide.html),
-and
-[Google's HTML/CSS Style Guide](https://google.github.io/styleguide/htmlcssguide.xml),
-with the following exceptions.
-
-* Rules related to Google Closure and JSDoc do not apply.
-* Modern Web Platform and JavaScript features should be preferred to legacy
- constructs that target old browsers.
-* Per the JavaScript guide, new tests should also follow any per-project
- style guide, such as the
- [ServiceWorker Tests Style guide](https://www.chromium.org/blink/serviceworker/testing).
+Our [document on layout tests tips](./layout_tests_tips.md) summarizes the most
+important WPT guidelines and highlights some JavaScript concepts that are worth
+paying attention to when trying to infer style rules from existing tests. If
+you're unopinionated and looking for a style guide to follow, the document also
+suggests some defaults.
## JavaScript Tests
@@ -337,10 +115,8 @@
```html
<!doctype html>
-<meta charset="utf-8">
-<title>JavaScript: the true literal</title>
+<title>JavaScript: the true literal is immutable and equal to itself</title>
<link rel="help" href="https://tc39.github.io/ecma262/#sec-boolean-literals">
-<meta name="assert" value="The true literal is equal to itself and immutable">
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script>
@@ -375,9 +151,6 @@
Some points that are not immediately obvious from the example:
-* The `<meta name="assert">` describes the purpose of the entire file, and
- is not redundant to `<title>`. Don't add a `<meta name="assert">` when the
- information in the `<title>` is sufficient.
* When calling an `assert_` function that compares two values, the first
argument is the actual value (produced by the functionality being tested), and
the second argument is the expected value (known good, golden). The order
@@ -428,6 +201,11 @@
cannot be implemented using Web APIs. The WPT project covers some of these cases
via supplemental testing APIs.
+When writing tests that rely on supplemental testing APIs, please consider the
+cost and benefits of having the tests
+[gracefully degrade to manual tests](./layout_tests_with_manual_fallback.md) in
+the absence of the testing APIs.
+
*** promo
In many cases, the user gesture is not actually necessary. For example, many
event handling tests can use
@@ -488,98 +266,6 @@
[](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)
-### Manual Tests
-
-🚧 Whenever possible, tests that rely on (WPT's or Blink's) testing APIs
-should also be usable as
-[manual tests](http://testthewebforward.org/docs/manual-test.html). This makes
-it easy to debug the test, and to check whether our behavior matches other
-browsers.
-
-*** promo
-The recommendation to degrade to manual tests is still being discussed on
-[blink-dev](https://groups.google.com/a/chromium.org/d/topic/blink-dev/XsR6PKRrS1E/discussion).
-However, please keep in mind that a manual test can be debugged in the browser,
-whereas a test that does not degrade gracefully can only be debugged in the test
-runner. Fellow project members and future you will thank you for having your
-test work as a manual test.
-***
-
-Manual tests should minimize the chance of user error. This implies keeping the
-manual steps to a minimum, and having simple and clear instructions that
-describe all the configuration changes and user gestures that match the effect
-of the Blink-specific APIs used by the test.
-
-Below is an example of a fairly minimal test that uses a Blink-Specific API
-(`window.eventSender`), and gracefully degrades to a manual test.
-
-```html
-<!doctype html>
-<meta charset="utf-8">
-<title>DOM: Event.isTrusted for UI events</title>
-<link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
-<link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
-<meta name="assert"
- content="Event.isTrusted is true for events generated by user interaction">
-<script src="../../resources/testharness.js"></script>
-<script src="../../resources/testharnessreport.js"></script>
-
-<p>Please click on the button below.</p>
-<button>Click Me!</button>
-
-<script>
-'use strict';
-
-setup({ explicit_timeout: true });
-
-promise_test(() => {
- const button = document.querySelector('button');
- return new Promise((resolve, reject) => {
- const button = document.querySelector('button');
- button.addEventListener('click', (event) => {
- resolve(event);
- });
-
- if (window.eventSender) {
- eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
- eventSender.mouseDown();
- eventSender.mouseUp();
- }
- }).then((clickEvent) => {
- assert_true(clickEvent.isTrusted);
- });
-
-}, 'Click generated by user interaction');
-
-</script>
-```
-
-The test exhibits the following desirable features:
-
-* It has a second specification URL (`<link rel="help">`), because the paragraph
- that documents the tested feature (referenced by the primary URL) is not very
- informative on its own.
-* It links to the
- [WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F),
- rather than to a frozen version of the specification.
-* It contains clear instructions for manually triggering the test conditions.
- The test starts with a paragraph (`<p>`) that tells the tester exactly what to
- do, and the `<button>` that needs to be clicked is clearly labeled.
-* It disables the timeout mechanism built into `testharness.js` by calling
- `setup({ explicit_timeout: true });`
-* It checks for the presence of the Blink-specific testing APIs
- (`window.eventSender`) before invoking them. The test does not automatically
- fail when the APIs are not present.
-* It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
- to separate the test setup from the assertions. This is particularly helpful
- for manual tests that depend on a sequence of events to occur, as Promises
- offer a composable way to express waiting for asynchronous events that avoids
- [callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).
-
-Notice that the test is pretty heavy compared to a minimal JavaScript test that
-does not rely on testing APIs. Only use testing APIs when the desired testing
-conditions cannot be set up using Web Platform APIs.
-
### Text Test Baselines
By default, all the test cases in a file that uses `testharness.js` are expected
@@ -704,7 +390,6 @@
```html
<!doctype html>
-<meta charset="utf-8">
<link rel="match" href="ol-reversed-expected.html">
<ol reversed>
@@ -718,7 +403,6 @@
```html
<!doctype html>
-<meta charset="utf-8">
<ol>
<li value="3">A</li>
@@ -844,7 +528,6 @@
```html
<!doctype html>
-<meta charset="utf-8">
<style>
body { font: 10px Ahem; }
span::after {
