What we can learn
from losing tests
slideshare.net/willcritchlow
@willcritchlow
Will Critchlow
SearchPilot
Have you
ever…?
You cross over
(Find an amazing keyword
research insight)
Pull up
(Modify titles and meta data
to match search intent)
What we can learn from losing SEO tests
What we can learn from losing SEO tests
What we can learn from losing SEO tests
-27%
We’ll come back
to this
Tweet me: @willcritchlow #BrightonSEO
What about
this?
Request a change /
create a ticket
Engineering says
it’ll take
18 months
?
I feel like a lot of SEO
is asking “Will you
come on an
adventure?”
Tweet me: @willcritchlow #BrightonSEO
Will you come on an adventure?
We don’t know what works
Tweet me: @willcritchlow #BrightonSEO
Will you come on an adventure?
We don’t know what works
But it’s OK because our recommendations
don’t get implemented anyway
Tweet me: @willcritchlow #BrightonSEO
Men wanted for hazardous
journey. Low wages, bitter cold,
long hours of complete
darkness. Safe return doubtful.
Honour and recognition in
event of success.
Ernest Shackleton
Tweet me: @willcritchlow #BrightonSEO
What if there
were a better
way?
Tweet me: @willcritchlow #BrightonSEO
What’s the
benefit of SEO
testing?
SEO testing means you can tie individual on-site changes to organic search
performance (reference).
SEO testing means you can tie individual on-site changes to organic search
performance (reference).
SEO testing means you can tie individual on-site changes to organic search
performance (reference).
+14%
organic
traffic
Tweet me: @willcritchlow #BrightonSEO
How are we
running these
tests?
Tweet me: @willcritchlow #BrightonSEO
Flights
NYC
Search
LA
Search
Seattle
Search
Austin
Search
Boston
Search
Dallas
Search
Find a group of pages using the same template
Tweet me: @willcritchlow #BrightonSEO
Create a hypothesis you want to test
{{City}} Flights
Search
{{City}} Flights
Search
Current template Proposed template
Tweet me: @willcritchlow #BrightonSEO
SearchPilot creates test buckets
Dallas Flights
Search
Austin Flights
Search
Boston
Flights
Search
NYC Flights
Search
LA Flights
Search
Seattle
Flights
Search
Control Pages
Variant Pages
Dallas Flights
Search
Austin Flights
Search
Boston
Flights
Search
NYC Flights
Search
LA Flights
Search
Seattle
Flights
Search
Tweet me: @willcritchlow #BrightonSEO
Flights
NYC
Search
LA
Search
Seattle
Search
Austin Flights
Search
Boston
Flights
Search
Dallas Flights
Search
SearchPilot applies changes to the variant pages
Austin
Search
Boston
Search
Dallas
Search
Tweet me: @willcritchlow #BrightonSEO
Flights
NYC
Search
LA
Search
Seattle
Search
Austin
Search
Boston
Search
Dallas
Search
SearchPilot applies changes to the variant pages
Tweet me: @willcritchlow #BrightonSEO
Search engine A/B
testing is different
from user A/B testing
Key point
Tweet me: @willcritchlow #BrightonSEO
A B A B A B A B A B A B
Splitting pages not users
SEO Testing
VS
User Testing
Tweet me: @willcritchlow #BrightonSEO
Conversion rate
optimisation focuses
on converting more
of your existing
traffic (bottom of the
funnel)
Conversion Rate
Optimisation
Tweet me: @willcritchlow #BrightonSEO
Search engine
optimisation focuses
on getting additional
traffic to your
website (top of
funnel)
Conversion Rate
Optimisation
Search Engine
Optimisation
Tweet me: @willcritchlow #BrightonSEO
Full funnel testing
combines these
approaches to do
both at the same
time
Full Funnel Testing
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Let’s talk
business cases
Tweet me: @willcritchlow #BrightonSEO
Obvious
benefits!
Positive
Tweet me: @willcritchlow #BrightonSEO
Avoid losing
changes
Negative
Tweet me: @willcritchlow #BrightonSEO
Save time
and money
No benefit
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Save time
and money
No benefit
Avoid losing
changes
Negative
← we’re
going to
talk about
these
cases
Tweet me: @willcritchlow #BrightonSEO
Negative tests
Tweet me: @willcritchlow #BrightonSEO
#1 Titles!
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Video
and
slides
here
Tweet me: @willcritchlow #BrightonSEO
#2
Removing
terrible(*)
content
(*) not terrible, apparently
{{City}} Flights
Search
{{City}} Flights
Search
Control Variant
That text looks like it’s just for SEO. What if we remove it?
“I would think about what you
can do to make those pages
rank well without having to put
a giant paragraph of content
below the page.”
– John Mueller, Google
-4% organic traffic
Read the full “removing SEO
text” case study here
Tweet me: @willcritchlow #BrightonSEO
#3
Adding
timely offers
You cross over
(Find an amazing keyword
research insight)
People modify their
searches with “Easter”
(travel deals / flights)
-14%
-11%
Why?!
Read the full
“Easter” case study
here
Tweet me: @willcritchlow #BrightonSEO
#4
Avoiding
terrible(*) auto-
generated
snippets
(*) not terrible, apparently
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
-3%
Why?!
Read the full Google
meta description
rewriting case study
Tweet me: @willcritchlow #BrightonSEO
Inconclusive
tests
Tweet me: @willcritchlow #BrightonSEO
Alt attributes
#1
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Read the full
ALT attributes case
study here
Tweet me: @willcritchlow #BrightonSEO
E-A-T &
authorship
#2
Tweet me: @willcritchlow #BrightonSEO
E-A-T has been on our minds
recently
(ahem, “helpful” content)
Tweet me: @willcritchlow #BrightonSEO
You might get
“Authorship”
recommendations
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
Tweet me: @willcritchlow #BrightonSEO
So be cautious if
anyone says
“we just need to add
some authorship
signals to this page”
Tweet me: @willcritchlow #BrightonSEO
Read the full
authorship and E-A-T
case study here
ACTION: get access to all our test results:
www.searchpilot.com/newsletter
SEO
testing
Full
funnel
SEO
testing
Testing
cadence
Full
funnel
SEO
testing
Testing
cadence
Full
funnel
SEO
testing
Executives
Marketing
leaders
Marketing
team
Testing
cadence
Full
funnel
SEO
testing
We are celebrating test momentum and cadence
& winning tests
new pages
Modern on-
site SEO
should cut
out untested
changes and
focus on:
https://bit.ly/BrightonMoneyball
Things to DO
For everyone - including agencies and smaller websites
www.searchpilot.com/newsletter
Things to DO
For those of you who work in the SEO
equivalent of the NBA:
Get me
to speak
to your
whole
team for
free
Tweet me: @willcritchlow #BrightonSEO
Thanks!
Follow @SearchPilot to take part in polls
Get on the email list at www.searchpilot.com
Ask me questions: @willcritchlow

More Related Content

PDF
The Ultimate Maturity Audit _ Brighton SEO.pdf
PPTX
Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...
PDF
Probabilistic Thinking in SEO - BrightonSEO October 2022
PPTX
I Am A Donut: How to Avoid International SEO Mistakes
PDF
BrightonSEO 2022.pdf
PDF
How to produce great multilingual content, even when you can't read it | Laur...
PPTX
How SEO changes, as we say bye bye to cookies
PDF
Agile SEO: Prioritise SEO Activities with Cadence and Risk Radius
The Ultimate Maturity Audit _ Brighton SEO.pdf
Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...
Probabilistic Thinking in SEO - BrightonSEO October 2022
I Am A Donut: How to Avoid International SEO Mistakes
BrightonSEO 2022.pdf
How to produce great multilingual content, even when you can't read it | Laur...
How SEO changes, as we say bye bye to cookies
Agile SEO: Prioritise SEO Activities with Cadence and Risk Radius

What's hot (20)

PDF
How to get more traffic with less content - BrightonSEO
PPTX
BrightonSEO - NLP for SEOs - How to optimise your content for BERT.pptx
PPTX
How to leverage indexation tracking to monitor issues and improve performance
PPTX
Data Pitfalls - Brighton SEO - Katie Swann.pptx
PPTX
BrightonSEO - Master Crawl Budget Optimization for Enterprise Websites
PPTX
Data-driven SEO & content strategy to reduce your customer acquisition costs
PPTX
EAT: Have We Been Looking At It Backwards
PDF
Brighton SEO 2023 - ML Lessons For Total Search.pdf
PPTX
Can you trust AI with your content?
PDF
Core Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdf
PPTX
Shining a light on the dark funnel
PDF
Networking for SEOs (and why it matters)
PDF
BrightonSEO October 2022 - Log File Analysis - Steven van Vessum.pdf
PDF
How to Incorporate ML in your SERP Analysis, Lazarina Stoy -BrightonSEO Oct, ...
PDF
How to control googlebot
PPTX
I Am A Donut - How To Avoid International SEO Mistakes
PDF
Using Search Intent in our Link Building Efforts
PPTX
Why your tech optimisations are still sat in the backlog
PDF
Improving Crawling and Indexing using Real-Time Log File Insights
PPTX
Lucy Dodds - BrightonSEO Autumn 22
How to get more traffic with less content - BrightonSEO
BrightonSEO - NLP for SEOs - How to optimise your content for BERT.pptx
How to leverage indexation tracking to monitor issues and improve performance
Data Pitfalls - Brighton SEO - Katie Swann.pptx
BrightonSEO - Master Crawl Budget Optimization for Enterprise Websites
Data-driven SEO & content strategy to reduce your customer acquisition costs
EAT: Have We Been Looking At It Backwards
Brighton SEO 2023 - ML Lessons For Total Search.pdf
Can you trust AI with your content?
Core Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdf
Shining a light on the dark funnel
Networking for SEOs (and why it matters)
BrightonSEO October 2022 - Log File Analysis - Steven van Vessum.pdf
How to Incorporate ML in your SERP Analysis, Lazarina Stoy -BrightonSEO Oct, ...
How to control googlebot
I Am A Donut - How To Avoid International SEO Mistakes
Using Search Intent in our Link Building Efforts
Why your tech optimisations are still sat in the backlog
Improving Crawling and Indexing using Real-Time Log File Insights
Lucy Dodds - BrightonSEO Autumn 22
Ad

Similar to What we can learn from losing SEO tests (20)

PDF
BrightonSEO - SearchPilot - Will Critchlow - Lessons from 100s of SEO tests
PDF
3 New Techniques for the Modern Age of SEO
PDF
The inbounder London - 2. May 2017 Tom Anthony
PPT
Social Websites And Seo Social Dev Camp Chicago2008 By John Fairley
PDF
Best practice is not enough #brightonSEO @Linkdex #SEOnow stage
PDF
brightonSEO - Metehan Yesilyurt - Generative AI & GEO: the new SEO race and h...
PDF
SEO Split-Testing - Why and How
PDF
SearchLove London 2016 | Tom Anthony | SEO Split-Testing - How You can Run Te...
PPTX
Social Proof Landing Page Conversion Optimization
PPTX
Social Proof Tips to Boost Landing Page Conversions
PDF
The Toughest Battles in SEO are Internal - Rachel Costello, Technical SEO, De...
PDF
Viral Videos | Marketing Strategies | Marketing | Content Marketing | SEO
PPT
SEO & Social Media - True Link Love
PDF
The Value of Featured Snippets (BrightonSEO 2023).pdf
PDF
SEO Tests on Big Sites & Small - What Etsy, Pinterest and Others Can Teach Us
PDF
Seo advice home page - swimming pools
PPTX
Scott Cowley - Fall Stukent Digital Summit
PDF
BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...
PDF
Save a Floundering Digital Marketing Campaign
PPTX
BrightonSEO - SearchPilot - Will Critchlow - Lessons from 100s of SEO tests
3 New Techniques for the Modern Age of SEO
The inbounder London - 2. May 2017 Tom Anthony
Social Websites And Seo Social Dev Camp Chicago2008 By John Fairley
Best practice is not enough #brightonSEO @Linkdex #SEOnow stage
brightonSEO - Metehan Yesilyurt - Generative AI & GEO: the new SEO race and h...
SEO Split-Testing - Why and How
SearchLove London 2016 | Tom Anthony | SEO Split-Testing - How You can Run Te...
Social Proof Landing Page Conversion Optimization
Social Proof Tips to Boost Landing Page Conversions
The Toughest Battles in SEO are Internal - Rachel Costello, Technical SEO, De...
Viral Videos | Marketing Strategies | Marketing | Content Marketing | SEO
SEO & Social Media - True Link Love
The Value of Featured Snippets (BrightonSEO 2023).pdf
SEO Tests on Big Sites & Small - What Etsy, Pinterest and Others Can Teach Us
Seo advice home page - swimming pools
Scott Cowley - Fall Stukent Digital Summit
BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...
Save a Floundering Digital Marketing Campaign
Ad

More from Will Critchlow (18)

PDF
SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...
PPTX
BrightonSEO: How to generate 8 million SEO test ideas - Will Critchlow
PDF
BrightonSEO - SearchPilot - Will Critchlow - When what’s good for users isn’t...
PDF
CRO and SEO together: what happens when what's good for users isn't good for ...
PDF
Gaps in the algorithm
PDF
Amazon Search Summit - the need for split testing in SEO
PDF
Lessons from SEO split-testing
PDF
SEO split tests you should run - Will Critchlow
PDF
What does Google want? Future of Digital Marketing 2015
PPTX
Will critchlow, meaningful metrics
PPTX
Internet trends for marketers
PPTX
Inbound on a shoestring - Searchlove Boston
PPTX
Black hat politics
PDF
Link building mediocre to great
PPTX
Common Technical Mistakes SMX Munich 2012
PPTX
Metrics that Matter
PPTX
Things I wish I'd known
PPTX
Why you should love seo
SEO in Turbulent Times - BrightonSEO San Diego 2023 - Will Critchlow, SearchP...
BrightonSEO: How to generate 8 million SEO test ideas - Will Critchlow
BrightonSEO - SearchPilot - Will Critchlow - When what’s good for users isn’t...
CRO and SEO together: what happens when what's good for users isn't good for ...
Gaps in the algorithm
Amazon Search Summit - the need for split testing in SEO
Lessons from SEO split-testing
SEO split tests you should run - Will Critchlow
What does Google want? Future of Digital Marketing 2015
Will critchlow, meaningful metrics
Internet trends for marketers
Inbound on a shoestring - Searchlove Boston
Black hat politics
Link building mediocre to great
Common Technical Mistakes SMX Munich 2012
Metrics that Matter
Things I wish I'd known
Why you should love seo

Recently uploaded (20)

PDF
It Takes a Village Campaign Plan Book; Sidra Medicine
PDF
digital marketing courses online with od
PPTX
AI Search Demystified: Future‑Proof Your Brand For AI‑Driven Queries
PPTX
Transform Your Business with Top Digital Marketing Services_EGlogics.pptx
PDF
AI powered Digital Marketing- How AI changes
DOCX
FCL vs. LCL Freight Forwarding An Ultimate Handbook for Logistics Experts.docx
PPTX
Digital-Marketing-Strategy-Trends-and-Best-Practices-for-2025 PPT3.pptx
PDF
Deloitte contemporary marketing-trends-2025.pdf
PDF
Chapter 8,9.pdfVGGGCFDRGFDXCRFTGDSEDSFCTGHNHGBVHG
PPTX
FINAL PPT strategic management lessons.pptx
PDF
EX Kathmandu _Kailash Mansarovar Yatra 2025 by Nagarjuna Travels.pdf
PDF
The B2B Startup Marketing Playbook - How To Build A Revenue-Generating B2B Ma...
PDF
Top-Rated Marketplaces to Buy Verified Cash App Accounts Safely.pdf
PDF
Social Media Marketing in 2025 blog 1 2.pdf
DOCX
IREV Platform: Future of Affiliate Marketing
PDF
Salmanubnu Zakariya P – Digital Marketer & Frontend Developer Portfolio
PDF
6 AI Marketing Myths That Are Slowing You Down & Draining Your Budgets
DOCX
Auctioneer project lead by Ali Hasnain jappa
PDF
CAP 9.- Building the Price Foundation.pdf
PDF
Search Central Live Deep Dive APAC 2025 LT
It Takes a Village Campaign Plan Book; Sidra Medicine
digital marketing courses online with od
AI Search Demystified: Future‑Proof Your Brand For AI‑Driven Queries
Transform Your Business with Top Digital Marketing Services_EGlogics.pptx
AI powered Digital Marketing- How AI changes
FCL vs. LCL Freight Forwarding An Ultimate Handbook for Logistics Experts.docx
Digital-Marketing-Strategy-Trends-and-Best-Practices-for-2025 PPT3.pptx
Deloitte contemporary marketing-trends-2025.pdf
Chapter 8,9.pdfVGGGCFDRGFDXCRFTGDSEDSFCTGHNHGBVHG
FINAL PPT strategic management lessons.pptx
EX Kathmandu _Kailash Mansarovar Yatra 2025 by Nagarjuna Travels.pdf
The B2B Startup Marketing Playbook - How To Build A Revenue-Generating B2B Ma...
Top-Rated Marketplaces to Buy Verified Cash App Accounts Safely.pdf
Social Media Marketing in 2025 blog 1 2.pdf
IREV Platform: Future of Affiliate Marketing
Salmanubnu Zakariya P – Digital Marketer & Frontend Developer Portfolio
6 AI Marketing Myths That Are Slowing You Down & Draining Your Budgets
Auctioneer project lead by Ali Hasnain jappa
CAP 9.- Building the Price Foundation.pdf
Search Central Live Deep Dive APAC 2025 LT

What we can learn from losing SEO tests