A/B Testing

An experiment compares your current page (the Control) with up to three variants. Each visitor sees one version, picked at random using the traffic split you set. Flint counts how many visitors on each version reach your goal and shows the results under Experiments in the Flint app.

You can test at two levels:

• •One element on a page. A headline, an image, a call to action, a form.
• •A whole page. A different layout or story against the page you have today.

Variants are pages on your Flint site, built in your design system. You don't need a separate testing tool.

1. 1.Ask Flint for the test, in the app or in Slack. Say which page, what to change, what counts as a win, and how to split traffic.
2. 2.Flint proposes the test: the variants, the goal, and the split. Nothing is built yet. Ask for changes until it's right, then tell Flint to go ahead.

3. 3.Flint builds the variants, publishes them, sets the goal, and launches the test in one go. In Slack, Flint asks for your approval before it launches.

4. 4.Open Experiments to follow the results. Your test is listed under Live.

If something stops the launch, the test stays under Draft and Flint tells you what's blocking it. See Troubleshooting below.

Be specific about the page, the change, and the goal. "Test a headline that leads with the outcome on /pricing, count clicks on Book a demo, 50/50" gets a better test than "test my pricing page."

• •Up to four pages per test. The Control plus three variants.
• •Any split, in whole percents. 50/50, 34/33/33, or 70/15/15 to keep most of your traffic on the current page. Every version gets at least 1%.
• •Short names. Variant names are up to 15 characters. Flint names each one after what's different from the Control, like "Outcome-led."

Each test has one goal. Setting a new goal replaces the old one. Pick the number your team reports on.

• •Clicks. Pick the buttons or links that count, like Book a demo or Start free trial. In the editor preview, the elements that count have a dashed green outline labeled Goal 1.
• •A page visit. Someone reaching a page on your Flint site you choose, like your form's /thank-you page. This is the best goal for a form, because it only counts completed submissions. For a checkout or booking page on another domain, count the click that leads there instead.
• •A custom event. Any event your site already sends from the browser to Flint's analytics, by name. It only counts on the experiment's pages. Server-side events don't count.
• •A conversion in another tool. For a purchase in Shopify or a lead in your CRM, Flint adds two parameters to the page URL: utm_extra3 identifies the experiment and utm_extra4 identifies the version (or control). Flint carries them onto the links on your page, so they reach your store or CRM with the order or lead. Ask Flint which values to look for.

The version parameters are added whether or not the visitor accepts cookies. Ask Flint to rename them or turn them off for your site.

Forms don't get the version automatically. To see which version a lead came from, ask Flint to add a field with the version to the form (by default, experiment_variant). Flint first asks which field name your CRM expects.

Experiments groups your tests under Draft, Live, and Completed. Open a test to see its status (for example, Running) and:

• •Versions. A screenshot, name, and path for each version.
• •Progress. When the test started, how many days it has run, and about how many days are left.
• •Results. For each version:
• •Visitors: people who saw that version.
• •Reached goal: visitors who completed the goal.
• •Goal rate: Reached goal divided by Visitors.
• •Lift: how much higher or lower a variant's goal rate is than the Control's.
• •Chance to win: how likely a variant is to beat the Control.
• •Goal rate over time. A chart with one line per version.

The Control shows a dash under Lift and Chance to win, because it's what the variants are measured against. With more than one variant, compare each variant to the Control, not to each other.

Goal rate, lift, and chance to win appear once each version has 100 visitors.

Flint works out how many visitors the test needs from the Control's goal rate, aiming to reliably detect a 30% difference. It starts estimating once the test has run for at least a day and the Control has 100 visitors and at least one conversion. Uneven splits take longer, because the smallest variant needs enough visitors too. Progress shows Enough data collected once the test reaches that number and every version has at least 100 visitors.

Flint doesn't end a test on its own. When there's enough data, ask Flint to wrap it up. Flint confirms the winner with you first.

• •If a variant wins, Flint replaces the Control page with the winning version, keeping the Control's URL, then publishes, stops the test, removes the other versions, and publishes again after cleanup.
• •If the Control wins, Flint stops the test, removes the variants, and publishes after cleanup.

A stopped test can't be started again. To keep testing, ask Flint for a new test. In Slack, Flint asks for your approval before it stops a test.

• •@Flint set up a new variant for my landing page and run an A/B test against it
• •@Flint test three new headlines on /pricing against the current one, with 70% of traffic on the current page
• •@Flint make the goal visits to /thank-you
• •@Flint how is my A/B test going?
• •@Flint the test on /pricing has enough data. Ship the winner.

Already using Optimizely, VWO, Statsig, or another platform? You can keep using it with Flint pages. Add the platform's script under Site Settings > Custom Code in the Head Scripts field, then ask Flint to build the variants and tag the elements you're testing. See A/B Testing with External Tools or the setup guides:

• •A/B Testing with VWO
• •A/B Testing with Optimizely
• •A/B Testing with Statsig

LaunchDarkly, Eppo, AB Tasty, and Adobe Target follow the same pattern.

To test ads rather than pages, use Google Ads experiments. More on Google's Ad Experiment page.

Progress shows "Estimating duration." Flint needs the test to have run for at least a day, and the Control to have 100 visitors and at least one conversion.

Goal rate, lift, and chance to win show a dash. Each version needs 100 visitors. Check back once the test has run longer.

Flint says the test can't launch. Flint tells you which of these is blocking it:

• •No goal is set. Tell Flint what counts as a win.
• •The setup has errors. Flint lists them. Fix them in the same chat.
• •The test already ended. A stopped test can't be started again. Ask Flint for a new one.
• •Another test is running on that page. Wait for it to finish, or end it first.
• •Site setup or results tracking is missing. Contact support.

Your store or CRM doesn't show which version a visitor saw. The version parameters are added when the visitor lands on the experiment page with a full page load, not when they click through to it from another page on your site without a full reload. They are added only for the first experiment on a page. Forms don't carry the version automatically. Ask Flint to add a field with the version to the form (by default, experiment_variant); Flint first asks which field name your CRM expects. Ask Flint to check that the links on each version carry the parameters.

• •Analytics & Tag Manager
• •Forms
• •Query Parameters
• •Cookie Consent & CMP Integration
• •Publishing
• •Flint in Slack