blob: 99dd0e5f3c1eef3df3fc4029b5b0b77c8e7291ea [file] [log] [blame] [view]
robliao2ff53322016-09-23 19:28:241# Field Trial Testing Configuration
2
ishermana509c1852017-03-23 21:11:353This directory contains the `fieldtrial_testing_config.json` configuration file,
4which is used to ensure test coverage of active field trials.
robliao2ff53322016-09-23 19:28:245
ishermana509c1852017-03-23 21:11:356For each study, the first available experiment after platform filtering is used
7as the default experiment for Chromium builds. This experiment is also used for
8perf bots and browser tests in the waterfall.
robliao2ff53322016-09-23 19:28:249
ishermana509c1852017-03-23 21:11:3510> Note: This configuration applies specifically to Chromium developer builds.
11> Chrome branded / official builds do not use these definitions.
robliao2ff53322016-09-23 19:28:2412
Alexei Svitkine3126867622020-12-02 23:03:0213> Note: Non-developer builds of Chromium (for example, non-Chrome browsers,
14> or Chromium builds provided by Linux distros) should disable the testing
15> config via the GN flag `fieldtrial_testing_like_official_build=true`.
16
Hiroshige Hayashizakiac2fc0c2018-12-03 19:20:5017> Note: This configuration is NOT used for content_browsertests or other test
18> targets based on content_shell.
19
robliao2ff53322016-09-23 19:28:2420## Config File Format
21
22```json
23{
24 "StudyName": [
25 {
26 "platforms": [Array of Strings of Valid Platforms for These Experiments],
27 "experiments": [
28 {
29 "//0": "Comment Line 0. Lines 0-9 are supported.",
30 "name": "ExperimentName",
31 "params": {Dictionary of Params},
32 "enable_features": [Array of Strings of Features],
33 "disable_features": [Array of Strings of Features]
34 },
35 ...
36 ]
37 },
38 ...
39 ],
40 ...
41}
42```
43
44The config file is a dictionary at the top level mapping a study name to an
Carlos Knippschilde12741b2019-12-12 01:26:2345array of *study configurations*. The study name in the configuration file
46**must** match the FieldTrial name used in the Chromium client code.
ishermana509c1852017-03-23 21:11:3547
48> Note: Many newer studies do not use study names in the client code at all, and
49> rely on the [Feature List API][FeatureListAPI] instead. Nonetheless, if a
Carlos Knippschilde12741b2019-12-12 01:26:2350> study has a server-side configuration, the study `name` specified here
51> must still match the name specified in the server-side configuration; this is
52> used to implement sanity-checks on the server.
robliao2ff53322016-09-23 19:28:2453
54### Study Configurations
55
ishermana509c1852017-03-23 21:11:3556Each *study configuration* is a dictionary containing `platforms` and
robliao2ff53322016-09-23 19:28:2457`experiments`.
58
ishermana509c1852017-03-23 21:11:3559`platforms` is an array of strings, indicating the targetted platforms. The
Dominik Röttsches1243e0f2019-07-08 18:49:5660strings may be `android`, `android_webview`, `chromeos`, `ios`, `linux`, `mac`,
61or `windows`.
robliao2ff53322016-09-23 19:28:2462
63`experiments` is an array containing the *experiments*.
64
ishermana509c1852017-03-23 21:11:3565The converter uses the `platforms` array to determine which experiment to use
66for the study. The first experiment matching the active platform will be used.
67
68> Note: While `experiments` is defined as an array, currently only the first
69> entry is used*\**. We would love to be able to test all possible study
70> configurations, but don't currently have the buildbot resources to do so.
71> Hence, the current best-practice is to identify which experiment group is the
72> most likely candidate for ultimate launch, and to test that configuration. If
73> there is a server-side configuration for this study, it's typically
74> appropriate to copy/paste one of the experiment definitions into this file.
75>
76> *\**
77> <small>
78> Technically, there is one exception: If there's a forcing_flag group
79> specified in the config, that group will be used if there's a corresponding
80> forcing_flag specified on the command line. You, dear reader, should
81> probably not use this fancy mechanism unless you're <em>quite</em> sure you
82> know what you're doing =)
83> </small>
robliao2ff53322016-09-23 19:28:2484
85### Experiments (Groups)
86Each *experiment* is a dictionary that must contain the `name` key, identifying
Ilya Sherman84225992020-02-28 00:39:2187the experiment group name.
ishermana509c1852017-03-23 21:11:3588
Ilya Sherman84225992020-02-28 00:39:2189> Note: Studies should typically use the [Feature List API][FeatureListAPI]. For
90> such studies, the experiment `name` specified in the testing config is still
91> required (for legacy reasons), but it is ignored. However, the lists of
92> `enable_features`, `disable_features`, and `params` **must** match the server
93> config. This is enforced via server-side Tricorder checks.
94>
95> For old-school studies that do check the actual experiment group name in the
96> client code, the `name` **must** exactly match the client code and the server
97> config.
robliao2ff53322016-09-23 19:28:2498
Jonathan Backer60ac0ec2020-06-17 15:40:5399The remaining keys -- `enable_features`, `disable_features`, `min_os_version`,
100and `params` -- are optional.
robliao2ff53322016-09-23 19:28:24101
robliao2ff53322016-09-23 19:28:24102`enable_features` and `disable_features` indicate which features should be
ishermana509c1852017-03-23 21:11:35103enabled and disabled, respectively, through the
104[Feature List API][FeatureListAPI].
105
Jonathan Backer60ac0ec2020-06-17 15:40:53106`min_os_version` indicates a minimum OS version level (e.g. "10.0.0") to apply
107the experiment. This string is decoded as a `base::Version`. The same version is
108applied to all platforms. If you need different versions for different
109platforms, you will need to use different studies.
110
Ilya Sherman84225992020-02-28 00:39:21111`params` is a dictionary mapping parameter name to parameter value.
112
ishermana509c1852017-03-23 21:11:35113> Reminder: The variations framework does not actually fetch any field trial
114> definitions from the server for Chromium builds, so any feature enabling or
115> disabling must be configured here.
116
117[FeatureListAPI]: https://cs.chromium.org/chromium/src/base/feature_list.h
robliao2ff53322016-09-23 19:28:24118
119#### Comments
120
121Each experiment may have up to 10 lines of comments. The comment key must be of
122the form `//N` where `N` is between 0 and 9.
123
124```json
125{
126 "AStudyWithExperimentComment": [
127 {
Paul Millerad77b7892018-07-11 20:07:43128 "platforms": ["chromeos", "linux", "mac", "windows"],
robliao2ff53322016-09-23 19:28:24129 "experiments": [
130 {
131 "//0": "This is the first comment line.",
132 "//1": "This is the second comment line.",
133 "name": "DesktopExperiment"
134 }
135 ]
136 }
137 ]
138}
139```
140
141### Specifying Different Experiments for Different Platforms
142Simply specify two different study configurations in the study:
143
144```json
145{
146 "DifferentExperimentsPerPlatform": [
147 {
Paul Millerad77b7892018-07-11 20:07:43148 "platforms": ["chromeos", "linux", "mac", "windows"],
robliao2ff53322016-09-23 19:28:24149 "experiments": [{ "name": "DesktopExperiment" }]
150 },
151 {
152 "platforms": ["android", "ios"],
153 "experiments": [{ "name": "MobileExperiment" }]
154 }
155 ]
156}
157```
158
159## Presubmit
160The presubmit tool will ensure that your changes follow the correct ordering and
161format.