SlideShare a Scribd company logo
experience
Advanced Experimentation Workshop
James Spittal, CEO, Web Marketing ROI
Holly Joshi, Sr. Customer Success Manager, Optimizely
Mansoor Omar, Technical Support Engineer, Optimizely
1. Tackling data problems
• Misconceptions/theories
• Data between GA and Optimizely
• How to ensure what you’re seeing is accurate
2. Overcoming common challenges faced operationally
• Writing regular expressions
• Cross-domain Optimizely
3. Making your campaign development faster and more efficient
• Optimizely Extensions
• Reducing flickering
Agenda
experience
CRO/MarTech Leadership = GSD
experience
experience
Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely
experience
All About The Data
What Optimizely measures and integrating with your data systems
1. Resolving the misconceptions
2. Implementation is the solution
3. Get the most common/most critical
right
Optimizely Data
experience
73%
… the percentage of Google Analytics implementations that are wrong (leading to
incorrect traffic numbers, incorrect revenue, etc).
Source: https://econsultancy.com/blog/10925-80-of-online-retailers-are-using-google-
analytics-incorrectly
Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely
Apples & Oranges
experience
If you’re not comparing the same thing –
you’re not going to see the same numbers.
Apples & Oranges
experience
Google Analytics Session Time-Outs (and
Optimizely End User Id)
By default, a Google Analytics session
expires after 30 minutes of user inactivity.
This is entirely different to how Optimizely
tracks a “unique visitor” (tied to Optimizely
End User Id cookie, 10 year expiry).
experience
Impressions (new Optimizely measurement
model) – How it actually works
Optimizely logs an event whenever an experiment activates (this can be seen as a ‘decision’
under the hood as an XMLHTTPRequest)
The decision carries a few pieces of information, including:
Visitor ID
The ID of the experiment / campaign which is activating
The ID of the variation the visitor is allocated to
UUID: A unique decision ID (used for deduplication)
Again, decision events are generated whenever an experiment activates, even if the visitor
has been targeted by the same experiment previously. So in a non-technical sense, you can
imagine that a decision is generated each time an experience is delivered to a visitor.
Impressions (new Optimizely measurement
model)
select
count(distinct(UUID))
from
decisions;
This deduplication on UUID is important because Optimizely clients sometimes send multiple
decisions for the same experiment activation (this is due to the batching behavior of the web
snippet).
Typically, impressions are allocated on an annual basis. So we just count up the number of
unique decision events, subtract them from the annual quota, and voila, you've got your
"impressions remaining" count for the year.
experience
Optimizely Results Screen Example
Pages misfiring/misactivating
experience
Audience Conditions – Custom JavaScript
experience
QA everything (DIY)
1. Make a spreadsheet of all of your
events, pages and audiences
2. Create a QA campaign for each, to verify
it is triggering when intended and not
triggering when it shouldn’t.
experience
Optimizely Accelerate
experience
Optimizely Accelerate
Total cost around $15k AUD, includes full events/pages/URLs/audiences QA and campaign build,
delivered by approved Optimizely Solutions Partners.
experience
experience
URLs
Custom, Cross-Domain, Multi-Page
1. Regular expressions (versus Simple
Match, Exact Match and Substring Match)
2. Multi-page tests
3. Making Optimizely work cross-domain
(and/or cross-project)
Pages & URLs
experience
1. Matching too freely
2. ‘example.com’ Matches (Substring) for:
yourdomain.com.au/example.com
Common ‘Pages’ mistakes
experience
Regular Expressions Cheat Sheet
experience
Regular Expressions Practice
1. This makes Optimizely cross-domain
work
2. In short - Injects an iFrame that allows
local storage/cookie to traverse
3. Put waitForOriginSync into your Project
JS
WaitForOriginSync
experience
WaitForOriginSync
experience
experience
Campaign Development
Extensions, Efficiencies and Performance
1. Extensions = more campaigns, faster
• Extensions 101
• Why extensions can be tricky
• Extension of extensions
2. Experiment flickering minimisation
Campaign Development
experience
Extensions
Extensions are extendable!
experience
1. Templates, modularity, performance,
strategy, operational savings…
2. Object- orientated
101
experience
1. How many input fields should we have?
2. Why is my extension not showing up/not
accessible?
3. Where did my extension go?
4. Data drops out of an extension
Challenges
experience
1. Translation
[:en]
Hello World
[:zh]
(Traditional Chinese)
2. Markdown
3. Extending extensions
Cool Use Cases
experience
Page Flash/Flickering
https://media.giphy.com/media/wdh1SvEn0E06I/giphy.g
if
Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely
1. WaitForElement & ES6
2. WaitUntil
3. See the Optimzely Developer Docs
4. Async vs. Sync
Document object model
challenges
experience
jQuery – what you need to know about
Optimizely X
Var $ = window.optimizely.get(‘jQuery’) [Optimizely jQuery] when
instrumenting jQuery in your Optimizely campaigns goes wrong versus
var $ = window.jQuery;
var $ = window.jQuery || window.optimizely.get(‘jQuery’);
experience
1. Use ‘utils’ (WaitForElement, etc)
2. Use ES6/Native JavaScript
3. Asynchronous versus synchronous
Most important things to know
experience
JavaScript Hell
Asynchronous versus synchronous
Don’t always believe what Google
PageSpeed Insights tells you 
experience
Document Object Model
experience
experience
Takeaways
Recap & Resources
1. If you need help with data analysis
problems or auditing your GA data, speak
to an Optimizely Solutions Partner or
GACP
2. Manny and Holly are available for on-
demand Optimizely support for Optimizely
customers
Next steps
experience
experience
Questions?
james@webmarketingroi.com.au
James Spittal
THANK YOU

More Related Content

PPTX
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Optimizely
 
PPTX
Triple Your Experiment Velocity by Integrating Optimizely with Your Data Ware...
Optimizely
 
PDF
What Your Customers Really Do Online: 5 Ways to Remove the Guesswork
Optimizely
 
PPTX
Testing Your Testing Program
Jill Martay
 
PPTX
Testing Across the Enterprise: How Cox Automotive Scales Experimentation to M...
Optimizely
 
PDF
Maximizing Optimizely Website Performance
Optimizely
 
PPTX
How to Reduce Churn with Better Product Adoption
Amity
 
PPTX
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
New Delhi Salesforce Developer Group
 
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Optimizely
 
Triple Your Experiment Velocity by Integrating Optimizely with Your Data Ware...
Optimizely
 
What Your Customers Really Do Online: 5 Ways to Remove the Guesswork
Optimizely
 
Testing Your Testing Program
Jill Martay
 
Testing Across the Enterprise: How Cox Automotive Scales Experimentation to M...
Optimizely
 
Maximizing Optimizely Website Performance
Optimizely
 
How to Reduce Churn with Better Product Adoption
Amity
 
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
New Delhi Salesforce Developer Group
 

What's hot (19)

PDF
Definition of A/B testing and Case Studies by Optimizely
RusseWeb
 
PPTX
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely
 
PDF
Run High Impact Experimentation with High-quality Customer Discovery
Optimizely
 
PDF
Optimizely Workshop: Take Action on Results with Statistics
Optimizely
 
PPTX
Metrics that Matter: Measure Experiment Impact with Optimizely's Data Platform
Optimizely
 
PPTX
Performance as a feature - Scale conf
John Clegg
 
PDF
Intuit - How to Scale Your Experimentation Program
Optimizely
 
PPTX
Program Management 101: Best Practices from Optimizely-on-Optimizely
Optimizely
 
PDF
Vwo talk 3 17-21
VWO
 
PDF
The real c in cro is communication
VWO
 
PDF
Ff for product success
VWO
 
PDF
Test & Learn: Automation & Experimentation - In-Depth Discussion on Data-Driv...
Optimizely
 
PDF
MVP Design Hacks
Naresh Jain
 
PPTX
Lean Startup101 2014-05-07
Guita Gopalan
 
PPTX
Gamification - Compliance Training
Geniteam
 
PPT
Openbadges for e-assessment - e-assessment scotland 2013
stevesidaway
 
PPTX
Lean Startup Experimentation, outputs to outcomes, short version.
Andrew Rusling
 
PPTX
NIMIWIPitch
MosesFam
 
PPTX
Common TurningPoint Questions Webinar - July 15, 2014
Derrick
 
Definition of A/B testing and Case Studies by Optimizely
RusseWeb
 
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely
 
Run High Impact Experimentation with High-quality Customer Discovery
Optimizely
 
Optimizely Workshop: Take Action on Results with Statistics
Optimizely
 
Metrics that Matter: Measure Experiment Impact with Optimizely's Data Platform
Optimizely
 
Performance as a feature - Scale conf
John Clegg
 
Intuit - How to Scale Your Experimentation Program
Optimizely
 
Program Management 101: Best Practices from Optimizely-on-Optimizely
Optimizely
 
Vwo talk 3 17-21
VWO
 
The real c in cro is communication
VWO
 
Ff for product success
VWO
 
Test & Learn: Automation & Experimentation - In-Depth Discussion on Data-Driv...
Optimizely
 
MVP Design Hacks
Naresh Jain
 
Lean Startup101 2014-05-07
Guita Gopalan
 
Gamification - Compliance Training
Geniteam
 
Openbadges for e-assessment - e-assessment scotland 2013
stevesidaway
 
Lean Startup Experimentation, outputs to outcomes, short version.
Andrew Rusling
 
NIMIWIPitch
MosesFam
 
Common TurningPoint Questions Webinar - July 15, 2014
Derrick
 
Ad

Similar to Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely (20)

PDF
Let's make users happy
dpuiu
 
PDF
Designing intrapreneurship program with gamified startup toolkit
Corporate Startup Summit
 
PDF
UX Lead Product Management
John Wyatt
 
PPTX
An Experimentation Framework: How to Position for Triple Digit Growth
Optimizely
 
PDF
Mark Opanasiuk: How to leverage your work with a Product Mindset (UA)
Lviv Startup Club
 
PDF
BEST Practices - Testing & Optimization | Bredan Rendan
Caleb Whitmore
 
PDF
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
PRWD
 
PDF
The Leaders Guide to Getting Started with Automated Testing
James Briers
 
PPT
User Centered Design 101 Experience Dynamics Web Seminar 1229121780083642 3
Aravind Rajagopalan
 
PPT
1
cnche
 
PPT
1
chejn
 
PPTX
You_Exec_-_Product_Design_Strategy_Complete.pptx
saneralex
 
PPTX
Webinar: How to Conduct Unmoderated Remote Usability Testing
UserZoom
 
PPT
How to do usability testing and eye tracking
Objective Experience
 
PDF
Optimizely x Live Demo
Optimizely
 
PDF
How to build confidence in your release cycle
DiUS
 
PPTX
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
Craig Sullivan
 
PPTX
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Craig Sullivan
 
PPTX
Mastering Paid Search Automation
ROI Revolution
 
PPT
Risk Driven Testing
Jorge Boria
 
Let's make users happy
dpuiu
 
Designing intrapreneurship program with gamified startup toolkit
Corporate Startup Summit
 
UX Lead Product Management
John Wyatt
 
An Experimentation Framework: How to Position for Triple Digit Growth
Optimizely
 
Mark Opanasiuk: How to leverage your work with a Product Mindset (UA)
Lviv Startup Club
 
BEST Practices - Testing & Optimization | Bredan Rendan
Caleb Whitmore
 
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
PRWD
 
The Leaders Guide to Getting Started with Automated Testing
James Briers
 
User Centered Design 101 Experience Dynamics Web Seminar 1229121780083642 3
Aravind Rajagopalan
 
You_Exec_-_Product_Design_Strategy_Complete.pptx
saneralex
 
Webinar: How to Conduct Unmoderated Remote Usability Testing
UserZoom
 
How to do usability testing and eye tracking
Objective Experience
 
Optimizely x Live Demo
Optimizely
 
How to build confidence in your release cycle
DiUS
 
3 Optimisation Decks : WAW Copenhagen - 27 Feb 2013
Craig Sullivan
 
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Craig Sullivan
 
Mastering Paid Search Automation
ROI Revolution
 
Risk Driven Testing
Jorge Boria
 
Ad

Recently uploaded (20)

PDF
The Future of Mobile Is Context-Aware—Are You Ready?
iProgrammer Solutions Private Limited
 
PDF
Software Development Methodologies in 2025
KodekX
 
PPTX
The Future of AI & Machine Learning.pptx
pritsen4700
 
PPTX
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
PDF
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 
PDF
GDG Cloud Munich - Intro - Luiz Carneiro - #BuildWithAI - July - Abdel.pdf
Luiz Carneiro
 
PDF
How Open Source Changed My Career by abdelrahman ismail
a0m0rajab1
 
PDF
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
PPTX
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
PPTX
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
PDF
How ETL Control Logic Keeps Your Pipelines Safe and Reliable.pdf
Stryv Solutions Pvt. Ltd.
 
PDF
Trying to figure out MCP by actually building an app from scratch with open s...
Julien SIMON
 
PDF
Economic Impact of Data Centres to the Malaysian Economy
flintglobalapac
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PDF
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
PDF
Orbitly Pitch Deck|A Mission-Driven Platform for Side Project Collaboration (...
zz41354899
 
PDF
Tea4chat - another LLM Project by Kerem Atam
a0m0rajab1
 
PDF
AI-Cloud-Business-Management-Platforms-The-Key-to-Efficiency-Growth.pdf
Artjoker Software Development Company
 
PDF
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
PPTX
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
The Future of Mobile Is Context-Aware—Are You Ready?
iProgrammer Solutions Private Limited
 
Software Development Methodologies in 2025
KodekX
 
The Future of AI & Machine Learning.pptx
pritsen4700
 
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 
GDG Cloud Munich - Intro - Luiz Carneiro - #BuildWithAI - July - Abdel.pdf
Luiz Carneiro
 
How Open Source Changed My Career by abdelrahman ismail
a0m0rajab1
 
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
How ETL Control Logic Keeps Your Pipelines Safe and Reliable.pdf
Stryv Solutions Pvt. Ltd.
 
Trying to figure out MCP by actually building an app from scratch with open s...
Julien SIMON
 
Economic Impact of Data Centres to the Malaysian Economy
flintglobalapac
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
Orbitly Pitch Deck|A Mission-Driven Platform for Side Project Collaboration (...
zz41354899
 
Tea4chat - another LLM Project by Kerem Atam
a0m0rajab1
 
AI-Cloud-Business-Management-Platforms-The-Key-to-Efficiency-Growth.pdf
Artjoker Software Development Company
 
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 

Modernize 2018: Advanced Experimentation Workshop - Web Marketing ROI & Optimizely