User Experience Testing with A/B Variants

AB testing can easily manipulate decisions under the guise of being "data-driven" if they're not used correctly. Sometimes AB tests are used to go through the motions to validate predetermined decisions and signal to leadership that the company is "data-driven" more than they're used to actually determine the right decision. After all, it's tough to argue with "we ran an AB test!" It's ⚡️data science⚡️... It sounds good, right? But what's under the hood? Here are a few things that could be under the hood of a shiny, sparkly AB test that lacks statistics and substance: 1. Primary metrics not determined before starting the experiment. If you're choosing metrics that look good and support your argument after starting the experiment... 🚩 2. Not waiting for stat sig and making an impulsive decision🚩 AB tests can look pretty wild in the first few days... wait it out until you reach stat sig or the test stalls. A watched pot never boils. 3. Users not being split up randomly. This introduces bias in the experiment and can lead to Sample Mismatch Ratio which invalidates the results🚩 4. Not isolating changes. If you're changing a button color, adding a new feature, and adding a new product offering, how do you know which variable to attribute to the metric outcome?🚩 You don't. 5. User contamination. If a user sees both the control and the treatment or other experiments, they become contaminated and it becomes harder to interpret the results clearly. 🚩 6. Paying too much attention to secondary metrics. The more metrics you analyze, the more likely one will be stat sig by chance 🚩 If you determined them as secondary, treat them that way! 7. Choosing metrics not likely to reach a stat sig difference. This happens with metrics that likely won't change a lot from small changes (like expecting a small change to increase bottom funnel metrics, ex. conversion rates in SaaS companies)🚩 8. Not choosing metrics aligned with the change you're making and the business goal. If you're changing a button color, should you be measuring conversion or revenue 10 steps down the funnel?🚩 AB testing is really powerful when done well, but it can also be like a hamster on a wheel-- running but not getting anywhere new. Do you wanna run an AB test to make a decision or to look good in front of leadership?

Ever presented rock-solid research only to hear "Thanks, but we're going with our gut on this one"? Securing stakeholder buy-in is rarely about the quality of your work. It's about something deeper. When you’re dealing with a research trust gap, ask yourself 5 questions. 👽 Are you speaking alien to earthlings? When you say jargon like "double diamond" or "information architecture," your stakeholders hear gibberish. Business leaders didn't learn UX in business school—and most never will. Translate everything into business outcomes they understand. Revenue growth. Customer retention. Cost savings. Competitive advantage.  Speak their native language, not yours. ⏰ What keeps them awake at 3am? Behind every skeptical question is a personal fear. That product manager who keeps shooting down your findings? They're terrified of missing their KPIs and losing their bonus. Have honest conversations about what they're personally on the hook for delivering. Then show how your research helps them achieve exactly that. ❓Are you treating assumptions as facts? You might think you know what questions matter to your stakeholders. You're probably wrong. Before starting research, explicitly ask: "What questions do you need answered to make this decision?" Then design your research to answer exactly those questions. ⚒️ Are you dying on the hill of methodological purity? Sometimes you have 8 hours for research instead of 8 weeks. Being dogmatic about "proper" research methods doesn’t always pay off. Focus on outcomes over process. If quick-and-dirty gets reliable insights that drive decisions, embrace it. 🍽️ Are you force-feeding them a seven-course meal when they wanted a snack? Executives need 30-second summaries. Product managers need actionable findings. Junior team members need hands-on learning. Tailor your approach to each one. You can also use my stakeholder persona mapping template here: https://bit.ly/43R7wom What’s the best advice you’ve heard about dealing with skeptical stakeholders?

👀 Lessons from the Most Surprising A/B Test Wins of 2024 📈 Reflecting on 2024, here are three surprising A/B test case studies that show how experimentation can challenge conventional wisdom and drive conversions: 1️⃣ Social proof gone wrong: an eCommerce story 🔬 The test: An eCommerce retailer added a prominent "1,200+ Customers Love This Product!" banner to their product pages, thinking that highlighting the popularity of items would drive more purchases. ✅ The result: The variant with social proof banner underperformed by 7.5%! 💡 Why It Didn't Work: While social proof is often a conversion booster, the wording may have created skepticism or users may have seen the banner as hype rather than valuable information. 🧠 Takeaway: By removing the banner, the page felt more authentic and less salesy. ⚡ Test idea: Test removing social proof; overuse can backfire making users question the credibility of your claims. 2️⃣ "Ugly" design outperforms sleek 🔬 The test: An enterprise IT firm tested a sleek, modern landing page against a more "boring," text-heavy alternative. ✅ The Result: The boring design won by 9.8% because it was more user friendly. 💡 Why It Worked: The plain design aligned better with users needs and expectations. 🧠 Takeaway: Think function over flair. This test serves as a reminder that a "beautiful" design doesn’t always win—it’s about matching the design to your audience's needs. ⚡ Test idea: Test functional designs of your pages to see if clarity and focus drive better results. 3️⃣ Microcopy magic: a SaaS example 🔬 The test: A SaaS platform tested two versions of their primary call-to-action (CTA) button on their main product page. "Get Started" vs. "Watch a Demo". ✅ The result: "Watch a Demo" achieved a 74.73% lift in CTR. 💡 Why It Worked: The more concrete, instructive CTA clarified the action and benefit of taking action. 🧠 Takeaway: Align wording with user needs to clarify the process and make taking action feel less intimidating. ⚡ Test idea: Test your copy. Small changes can make a big difference by reducing friction or perceived risk. 🔑 Key takeaways ✅ Challenge assumptions: Just because a design is flashy doesn’t mean it will work for your audience. Always test alternatives, even if they seem boring. ✅ Understand your audience: Dig deeper into your users' needs, fears, and motivations. Insights about their behavior can guide more targeted tests. ✅ Optimize incrementally: Sometimes, small changes, like tweaking a CTA, can yield significant gains. Focus on areas with the least friction for quick wins. ✅ Choose data over ego: These tests show, the "prettiest" design or "best practice" isn't always the winner. Trust the data to guide your decision-making. 🤗 By embracing these lessons, 2025 could be your most successful #experimentation year yet. ❓ What surprising test wins have you experienced? Share your story and inspire others in the comments below ⬇️ #optimization #abtesting

founder learnings! part 8. A/B test math interpretation - I love stuff like this: Two members of our team (Fletcher Ehlers and Marie-Louise Brunet) - ran a test recently that decreased click-through rate (CTR) by over 10% - they added a warning telling users they’d need to log in if they clicked. However - instead of hurting conversions like you’d think, it actually increased them. As in - Fewer users clicked through, but overall, more users ended up finishing the flow. Why? Selection bias & signal vs. noise. By adding friction, we filtered out low-intent users—those who would have clicked but bounced at the next step. The ones who still clicked knew what they were getting into, making them far more likely to convert. Fewer clicks, but higher quality clicks. Here's a visual representation of the A/B test results. You can see how the click-through rate (CTR) dropped after adding friction (fewer clicks), but the total number of conversions increased. This highlights the power of understanding selection bias—removing low-intent users improved the quality of clicks, leading to better overall results.

🚨 Your A/B test results are not the real impact. A happy PM runs an A/B test → sees a +15% lift in revenue → scales the feature to all users → shares the big win in Slack 🎉 But… once the feature is fully rolled out, the KPI impact isn’t there. Why? Because test results often don’t reflect the true long-term effect. Here are a few reasons why this happens: 1️⃣ Confidence intervals matter → That “+15%” is actually a range. The lower bound might be close to zero. 2️⃣ Novelty effect → Users are excited at first, but the effect fades as they get used to the change. 3️⃣ Experiments aren’t additive → Three +15% lifts don’t stack to +45%. There’s a ceiling, and improvements often cannibalize each other. 4️⃣ Sample ≠ population → The test group might not represent your entire user base. For example, you have more high-intent users in the variant. 5️⃣ Time-to-KPI effects → We see that a lot, especially in conversion experiments. The experiment could improve the time to conversion, so when you close the experiment, it seems like you’re winning, but actually if you monitor the users a few days/weeks after the experiment ends, there are no differences in total conversions between the variant and the control. 6️⃣ Type I error → With P-value=0.05 (or worse, 0.1), there’s still a decent chance the “win” is a false positive. 👉 That’s why tracking post-launch impact is just as important as running the experiment itself. Methods like holdout groups, simple correlation tracking, or causal inference models (building synthetic control) help reveal the real sustained effect.

🟠 Most data scientists (and test managers) think explaining A/B test results is about throwing p-values and confidence intervals at stakeholders... I've sat through countless meetings where the room goes silent the moment a technical slide appears. Including mine. You know the moment when "statistical significance" and "confidence intervals" flash on screen, and you can practically hear crickets 🦗 It's not that stakeholders aren't smart. We are just speaking different languages. Impactful data people uses completely opposite approach. --- Start with the business question --- ❌ "Our test showed a statistically significant 2.3% lift..." ✅ "You asked if we should roll out the new recommendation model..." This creates anticipation and you may see the stakeholder lean forward. --- Size the real impact --- ❌ "p-value is 0.001 with 95% confidence..." ✅ "This change would bring in ~$2.4M annually, based on current traffic..." Numbers without context are just math. They can be in appendix or footnotes. Numbers tied to business outcomes are insights. These should be front and center. --- Every complex idea has a simple analogy --- ❌ "Our sample suffers from selection bias..." ✅ "It's like judging an e-commerce feature by only looking at users who completed a purchase..." --- Paint the full picture. Every business decision has tradeoffs --- ❌ "The test won", then end presentation ✅ Show the complete story - what we gained, what we lost, what we're still unsure about, what to watch post-launch, etc. --- This one is most important --- ✅ Start with the decision they need to make. Then only present the data that helps make **that** decision. Everything else is noise. The core principle at work? Think like a business leader who happens to know data science. Not a data scientist who happens to work in business. This shift in mindset changes everything. Are you leading experimentation at your company? Or wrestling with translating complex analyses into clear recommendations? I've been there. For 16 long years. In the trenches. Now I'm helping fellow data practitioners unlearn the jargon and master the art of influence through data. Because let's be honest - the hardest part of our job isn't running the analysis. It's getting others to actually use it.

Recently, someone shared results from a UX test they were proud of. A new onboarding flow had reduced task time, based on a very small handful of users per variant. The result wasn’t statistically significant, but they were already drafting rollout plans and asked what I thought of their “victory.” I wasn’t sure whether to critique the method or send flowers for the funeral of statistical rigor. Here’s the issue. With such a small sample, the numbers are swimming in noise. A couple of fast users, one slow device, someone who clicked through by accident... any of these can distort the outcome. Sampling variability means each group tells a slightly different story. That’s normal. But basing decisions on a single, underpowered test skips an important step: asking whether the effect is strong enough to trust. This is where statistical significance comes in. It helps you judge whether a difference is likely to reflect something real or whether it could have happened by chance. But even before that, there’s a more basic question to ask: does the difference matter? This is the role of Minimum Detectable Effect, or MDE. MDE is the smallest change you would consider meaningful, something worth acting on. It draws the line between what is interesting and what is useful. If a design change reduces task time by half a second but has no impact on satisfaction or behavior, then it does not meet that bar. If it noticeably improves user experience or moves key metrics, it might. Defining your MDE before running the test ensures that your study is built to detect changes that actually matter. MDE also helps you plan your sample size. Small effects require more data. If you skip this step, you risk running a study that cannot answer the question you care about, no matter how clean the execution looks. If you are running UX tests, begin with clarity. Define what kind of difference would justify action. Set your MDE. Plan your sample size accordingly. When the test is done, report the effect size, the uncertainty, and whether the result is both statistically and practically meaningful. And if it is not, accept that. Call it a maybe, not a win. Then refine your approach and try again with sharper focus.

Most teams are drowning in optimization test ideas... but starving for real impact. I've seen this pattern destroy more optimization programs than poor execution ever could. The problem isn't lack of creativity. It's lack of strategy. Before you run another A/B test, ask yourself four critical questions: ↳ Is this strategically important to your business goals? ↳ Are you confident the change won't harm the user experience? ↳ Can you reach statistical significance in a reasonable timeframe? ↳ Do you have the technical capability to execute properly? If any answer is "no," you have better options: ↳ De-prioritize non-strategic tests. Add them to your backlog for later consideration. ↳ Run rapid sentiment tests or task completion analysis for quick validation. Only commit to full experimentation when all four criteria align. Or implement proven solutions directly when you're confident in the outcome. This decision framework has helped our clients at The Good generate over $100 million in additional revenue by focusing their testing efforts where they matter most. Your optimization program isn't measured by how many tests you run. It's measured by how much value you create.

Stakeholders don’t hate UX. They hate UX language. After 9+ years of watching incredible work go unnoticed by the powers that be, I’m certain of this: The problem usually isn’t the research. Or the design. It’s the translation. We speak UX—they speak business. If you want your work to land, you have to bridge that gap. Here’s how: 1️⃣ Translate insights into impact ❌ “Our usability testing revealed friction in the checkout flow.” ✅ “We uncovered a bottleneck costing $50K/month in abandoned carts.” 2️⃣ Lead with the outcome, not the method Don’t open with how you ran the study. Open with what it means for the business: “This change could lift conversion 15%.” Then explain how you got there. 3️⃣ Use their success metrics UX metrics are for us. Execs want CAC, LTV, churn, retention. Frame your work in their language. 4️⃣ Show, don’t summarize Skip the 40-slide deck. Play a 90-second video of a user getting stuck. You don’t need buy-in when someone feels the pain. 5️⃣ Make it about them—not us ❌ “UX research shows…” ✅ “Your customers are telling us…” Same data. Different gravity. The best UX leaders I know? They’re translators first, designers second. They turn user frustration into business opportunity. Research findings into revenue forecasts. Because influence doesn’t come from pixels. It comes from speaking the right language. What’s your go-to phrase for getting stakeholder buy-in? Drop it below—someone may need it. #uxdesign #uxleadership #productstrategy ——— 👋 Hi, I’m Dane—I love sharing design tips + strategies. ❤️ Found this helpful? Dropping a like shows support. 🔄 Share to help others (& for easy access later). ➕ Follow for more like this in your feed every day.

How a mobile cart redesign increased transactions by 3.4% Problem: Checkout drop-off rates were killing mobile revenue. → The cart design was cluttered, unintuitive, and frustrating for users. → Visitors struggled to understand their next steps, leading to high abandonment rates. Solution: We did a deep dive into user behavior with: - Google Analytics: To identify friction points in the funnel. - HotJar heatmaps: To track user interactions and frustrations. - User Testing: To understand why visitors were dropping off. What we found: Visitors needed clearer CTAs, smoother layout, tap-friendly elements. We implemented a mobile-specific cart redesign with these improvements: Larger tap targets for easy navigation. Streamlined layout to reduce decision fatigue. Stronger calls-to-action to guide users through checkout. Testing Process: We A/B tested the revamped cart design against the original. - Audience: Mobile visitors. - Metric: Increase in visits to checkout. - Duration: Conducted over a statistically significant period. Results: The redesign delivered across all key metrics: - +8% lift in visits to checkout. - +3.4% increase in transactions. - $1.39 boost in revenue per visitor (RPV). Here’s how you can use this for your brand: Eliminate friction with clear pathways. Simplify deep-funnel elements for mobile users. Invoke the “Don’t Make Me Think” principle to guide users seamlessly to checkout.