robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 1 | # Field Trial Testing Configuration |
| 2 | |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 3 | This directory contains the `fieldtrial_testing_config.json` configuration file, |
| 4 | which is used to ensure test coverage of active field trials. |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 5 | |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 6 | For each study, the first available experiment after platform filtering is used |
| 7 | as the default experiment for Chromium builds. This experiment is also used for |
| 8 | perf bots and browser tests in the waterfall. |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 9 | |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 10 | > Note: This configuration applies specifically to Chromium developer builds. |
| 11 | > Chrome branded / official builds do not use these definitions. |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 12 | |
Alexei Svitkine | 312686762 | 2020-12-02 23:03:02 | [diff] [blame^] | 13 | > Note: Non-developer builds of Chromium (for example, non-Chrome browsers, |
| 14 | > or Chromium builds provided by Linux distros) should disable the testing |
| 15 | > config via the GN flag `fieldtrial_testing_like_official_build=true`. |
| 16 | |
Hiroshige Hayashizaki | ac2fc0c | 2018-12-03 19:20:50 | [diff] [blame] | 17 | > Note: This configuration is NOT used for content_browsertests or other test |
| 18 | > targets based on content_shell. |
| 19 | |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 20 | ## Config File Format |
| 21 | |
| 22 | ```json |
| 23 | { |
| 24 | "StudyName": [ |
| 25 | { |
| 26 | "platforms": [Array of Strings of Valid Platforms for These Experiments], |
| 27 | "experiments": [ |
| 28 | { |
| 29 | "//0": "Comment Line 0. Lines 0-9 are supported.", |
| 30 | "name": "ExperimentName", |
| 31 | "params": {Dictionary of Params}, |
| 32 | "enable_features": [Array of Strings of Features], |
| 33 | "disable_features": [Array of Strings of Features] |
| 34 | }, |
| 35 | ... |
| 36 | ] |
| 37 | }, |
| 38 | ... |
| 39 | ], |
| 40 | ... |
| 41 | } |
| 42 | ``` |
| 43 | |
| 44 | The config file is a dictionary at the top level mapping a study name to an |
Carlos Knippschild | e12741b | 2019-12-12 01:26:23 | [diff] [blame] | 45 | array of *study configurations*. The study name in the configuration file |
| 46 | **must** match the FieldTrial name used in the Chromium client code. |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 47 | |
| 48 | > Note: Many newer studies do not use study names in the client code at all, and |
| 49 | > rely on the [Feature List API][FeatureListAPI] instead. Nonetheless, if a |
Carlos Knippschild | e12741b | 2019-12-12 01:26:23 | [diff] [blame] | 50 | > study has a server-side configuration, the study `name` specified here |
| 51 | > must still match the name specified in the server-side configuration; this is |
| 52 | > used to implement sanity-checks on the server. |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 53 | |
| 54 | ### Study Configurations |
| 55 | |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 56 | Each *study configuration* is a dictionary containing `platforms` and |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 57 | `experiments`. |
| 58 | |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 59 | `platforms` is an array of strings, indicating the targetted platforms. The |
Dominik Röttsches | 1243e0f | 2019-07-08 18:49:56 | [diff] [blame] | 60 | strings may be `android`, `android_webview`, `chromeos`, `ios`, `linux`, `mac`, |
| 61 | or `windows`. |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 62 | |
| 63 | `experiments` is an array containing the *experiments*. |
| 64 | |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 65 | The converter uses the `platforms` array to determine which experiment to use |
| 66 | for the study. The first experiment matching the active platform will be used. |
| 67 | |
| 68 | > Note: While `experiments` is defined as an array, currently only the first |
| 69 | > entry is used*\**. We would love to be able to test all possible study |
| 70 | > configurations, but don't currently have the buildbot resources to do so. |
| 71 | > Hence, the current best-practice is to identify which experiment group is the |
| 72 | > most likely candidate for ultimate launch, and to test that configuration. If |
| 73 | > there is a server-side configuration for this study, it's typically |
| 74 | > appropriate to copy/paste one of the experiment definitions into this file. |
| 75 | > |
| 76 | > *\** |
| 77 | > <small> |
| 78 | > Technically, there is one exception: If there's a forcing_flag group |
| 79 | > specified in the config, that group will be used if there's a corresponding |
| 80 | > forcing_flag specified on the command line. You, dear reader, should |
| 81 | > probably not use this fancy mechanism unless you're <em>quite</em> sure you |
| 82 | > know what you're doing =) |
| 83 | > </small> |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 84 | |
| 85 | ### Experiments (Groups) |
| 86 | Each *experiment* is a dictionary that must contain the `name` key, identifying |
Ilya Sherman | 8422599 | 2020-02-28 00:39:21 | [diff] [blame] | 87 | the experiment group name. |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 88 | |
Ilya Sherman | 8422599 | 2020-02-28 00:39:21 | [diff] [blame] | 89 | > Note: Studies should typically use the [Feature List API][FeatureListAPI]. For |
| 90 | > such studies, the experiment `name` specified in the testing config is still |
| 91 | > required (for legacy reasons), but it is ignored. However, the lists of |
| 92 | > `enable_features`, `disable_features`, and `params` **must** match the server |
| 93 | > config. This is enforced via server-side Tricorder checks. |
| 94 | > |
| 95 | > For old-school studies that do check the actual experiment group name in the |
| 96 | > client code, the `name` **must** exactly match the client code and the server |
| 97 | > config. |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 98 | |
Jonathan Backer | 60ac0ec | 2020-06-17 15:40:53 | [diff] [blame] | 99 | The remaining keys -- `enable_features`, `disable_features`, `min_os_version`, |
| 100 | and `params` -- are optional. |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 101 | |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 102 | `enable_features` and `disable_features` indicate which features should be |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 103 | enabled and disabled, respectively, through the |
| 104 | [Feature List API][FeatureListAPI]. |
| 105 | |
Jonathan Backer | 60ac0ec | 2020-06-17 15:40:53 | [diff] [blame] | 106 | `min_os_version` indicates a minimum OS version level (e.g. "10.0.0") to apply |
| 107 | the experiment. This string is decoded as a `base::Version`. The same version is |
| 108 | applied to all platforms. If you need different versions for different |
| 109 | platforms, you will need to use different studies. |
| 110 | |
Ilya Sherman | 8422599 | 2020-02-28 00:39:21 | [diff] [blame] | 111 | `params` is a dictionary mapping parameter name to parameter value. |
| 112 | |
isherman | a509c185 | 2017-03-23 21:11:35 | [diff] [blame] | 113 | > Reminder: The variations framework does not actually fetch any field trial |
| 114 | > definitions from the server for Chromium builds, so any feature enabling or |
| 115 | > disabling must be configured here. |
| 116 | |
| 117 | [FeatureListAPI]: https://cs.chromium.org/chromium/src/base/feature_list.h |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 118 | |
| 119 | #### Comments |
| 120 | |
| 121 | Each experiment may have up to 10 lines of comments. The comment key must be of |
| 122 | the form `//N` where `N` is between 0 and 9. |
| 123 | |
| 124 | ```json |
| 125 | { |
| 126 | "AStudyWithExperimentComment": [ |
| 127 | { |
Paul Miller | ad77b789 | 2018-07-11 20:07:43 | [diff] [blame] | 128 | "platforms": ["chromeos", "linux", "mac", "windows"], |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 129 | "experiments": [ |
| 130 | { |
| 131 | "//0": "This is the first comment line.", |
| 132 | "//1": "This is the second comment line.", |
| 133 | "name": "DesktopExperiment" |
| 134 | } |
| 135 | ] |
| 136 | } |
| 137 | ] |
| 138 | } |
| 139 | ``` |
| 140 | |
| 141 | ### Specifying Different Experiments for Different Platforms |
| 142 | Simply specify two different study configurations in the study: |
| 143 | |
| 144 | ```json |
| 145 | { |
| 146 | "DifferentExperimentsPerPlatform": [ |
| 147 | { |
Paul Miller | ad77b789 | 2018-07-11 20:07:43 | [diff] [blame] | 148 | "platforms": ["chromeos", "linux", "mac", "windows"], |
robliao | 2ff5332 | 2016-09-23 19:28:24 | [diff] [blame] | 149 | "experiments": [{ "name": "DesktopExperiment" }] |
| 150 | }, |
| 151 | { |
| 152 | "platforms": ["android", "ios"], |
| 153 | "experiments": [{ "name": "MobileExperiment" }] |
| 154 | } |
| 155 | ] |
| 156 | } |
| 157 | ``` |
| 158 | |
| 159 | ## Presubmit |
| 160 | The presubmit tool will ensure that your changes follow the correct ordering and |
| 161 | format. |