Why Page Speed Isn't Enough - Tim Morrow - Velocity Europe 2012

Editor's Notes

• #2: My name is Tim Morrow I work at BetfairI’ve spoken in the past about Betfair’s focus on performance as we built our new website platform to support our Sports betting website.We defined a customer commitment, documented our target page load and put in place the measurements we needed to assert that we were meeting those commitments. Our average page load times dropped to 3 seconds or less.However, when we spoke to our customers, they kept telling us our site was slow. They chose to remain on our old site.Today I’m going to talk about what we learned from our customers and why measuring page speed isn’t enough.
• #3: Betfair is a betting company; but we’re different to most other betting companies.Our primary betting product is our betting exchange.It’s a market for matching customer bets that works like a financial exchange.Prices are constantly moving; customers can trade their positions before and during eventsWe have a broad customer base consisting of over 4m customers across 100s of countries and territoriesThey bet on a diverse range of events such as Horse Racing, Greyhounds, Soccer, Tennis, Golf, Cricket, DartsAnd they exhibit different behaviours. Our customers range from grandmothers placing a £5 bet on the Grand National to professional trading businesses
• #4: Just to provide the scale: as money flows through our exchange, prices are changing 10 times per secondAutomated trading tools and human customers take positions for and against outcomesAs events progress in-play, scores change and prices move as customers trade their positions>>> For example, during the US Open tennis final, £63m pounds was traded during the 5 hour match on our exchange>>> This summer during the Euro 2012 football competition £1bnpounds was tradedReferences: http://bit.ly/V1NQpqhttp://bit.ly/VeE3Ld
• #5: Last year we published our customer commitment. >>> We said“Our products will meet the needs of our customers by delivering a quick and responsive experience”From a website perspective, we defined our promise in terms of full page load timesWe added Keynote measures from 11 geographic locations with alerting in place and daily reportsWe use our private webpagetest.org instance to continuously measure full page load times in our test labWe instrumented our pages withJavascript to capture real user measures beaconed back to our analytics system
• #6: And by those measures we made good progress.Take a look at a 2 week front-end full page load measurement via a DSL Keynote agentEach line represents a different page in a 4-page user journey. They vary based on the complexity of the page.So in the first page, also our most complicated, we aren’t hitting our target 3 seconds, but other pages are significantly faster.We saw similar improvements from our RUM measurements too
• #7: We get feedback from customers via a number of mediums: They call us, respond to our on-line feedback form, they post in our forumsWhat did they have to say? There were many positive comments but there were equally as many comments complaining about it being slower. >>> Slower and harder to place bets>>> Same again>>> Old site faster to complete tasksBut we had empirical proof it was faster; where was the gap in our understanding?
• #8: So let me talk about the cost of this poor perceived performanceAs we built the site, we provided the ability for customers to revert to use our legacy site; there was an option to click a link and select Revert To Old>>> Click <<<Initially our opt-out rate was low; we started with Football as the only available sport, then rolled out other sports in waves, increasing the breadth of our product offering and encompassing a greater proportion of our customer baseBut, as we added those sportsthe number of customers choosing to opt-out increased; and those same customers chose to stay away even as we attempted to entice them back.We had delivered lots of improvements and new features on our new site. It was a (functionally) better product.So aside from the cost of maintaining two website products, there was a significant opportunity costs to have those customers continue to use our old site; it was imperative for us to understand the root issues and rectify them
• #9: So we wanted to understand our customers and what their behaviours areBetfair has a large and varied customer base; importantly our customers exhibit very different characteristics;we analyse our customer activity via our web analytics and data warehouse and segment them based on their value within our ecosystemWe found our higher value customers were opting out and a significantly greater rate than other customersSo we took a look at our highest value customers
• #10: So this first chart is showing the proportion of High Value customers (gold) to low value customers (blue).Our highest value segments make up 23% of visitors>>> Click <<<However, they place 58% of betsAgain high value customers are gold, low value customers in blueBy focusing on the high value customers we could make the most impactful improvements
• #11: We examined the behaviours of our highest value customers and identified four key tasks that they perform most often. Many of these tasks involve complex bet placements with dozens or hundreds of selections and a high degree of interaction with our siteWe developed automated measurement of these tasks using Selenium in a test labComparing them on our new site to the old site, we found the experience to be significantly worse.Let me illustrate that
• #12: Here’s a chart showing the timing of a complex task – typical of a high value customers.The Y-AXIS is task completion time in seconds when executed via a Selenium scriptBlue is our old site;Gold is our new site;clearly there was a massive gap.>>> Click to see steps <<<So we broke the task down into the individual steps that customers would take to complete this taskThe data provides a stark illustration of the problem; there was something fundamentally wrong on our new experienceBut because we hadn’t been measuring it we were oblivious – aside from the vocal customers who had taken the time to tell us our site was slow
• #13: We put together a team to review each journey and recommend improvements.They quicklyiterated and measured the performance at each iteration; we quickly achieved parity between new & oldThis graph simply shows the improvement to the overall task completion time measured using our automated scripts as we iteratedThis wasn’t some fundamental architectural problem – its what happens when you don’t focus on key areas; you layer feature upon feature; when you pace and don’t measureSimply put, we took our eye off the ball
• #14: Next, we looked beyond our simplistic 4-page journeys that we’d worked so hard to optimize.Again, we focused on our high value customer segments and examined their sessions and click patterns.
• #15: Our highest value customers spend a significant amount of time on site; 80% more than lower value customer segments>>> They also visit 30% more pages, on averageSo we knew fast navigation was important; These customers navigate quickly between markets and place numerous bets per market while in-playIn this scenario, even 3 second full page loads would feel slow
• #16: We built our site in a loosely coupled modular fashion but launched only with server-side rendering of complete pages.Our architecture consists of individual modules that render themselves when the data they need is available.>>> Click 6 times to show modules <<<Each of these is a module that requests its own data and can be rendered in isolation on the server-sideWe exploited our architecture to add a navigation feature that only updates the bits of the page that change via Ajax;we still server-side render the page modules, but insert them into the page>>> Click <<<So if these modules are the only ones that need to change, that’s all that changesThe calls are lightweight and load pretty quicklyThe rest of the page furniture remains static
• #17: Let me illustrate this Ajax Architecture improvementI’m going to play a 25 second video in which I’llto navigate through 21 horse racing markets.The section in the middle – that provides the event header, runners, live prices – is different for each market.Keep your eye on the page furniture – the header, the left-hand menu, the right-hand video module>>> Click to Play <<<The slowest part is the human clicking through the menu. That’d be me.With 3 seconds page loads, this would have taken significantly longerWe’ve still got some work to do; currently at around 200ms, we’d like to get it down to 100ms – so they load in the blink of an eye
• #19: Know your customersUnderstand how your most valuable customers interact with your siteSpeak to them; watch them use your product; ask their opinion; listen to their feedbackUse your analytics data to understand customer behaviours
• #20: Page speed isn’t enoughMeasure what matters for the customers that matterCustomers don’t come to your site to load the pages;they come to browse, to discover, to discuss, to build, to buy, to… betIdentify key areas of concern and optimise transactional flowsLook to optimize the entire experience; consider implementation approaches such as Ajax architecture
• #21: Always remember; you truly do get what you measure