SlideShare a Scribd company logo
Continuous Testing
at Scale
Gergely Orosz, Engineering Manager
@GergelyOrosz
8 May 2018
● Engineering manager @Uber, in Amsterdam
● 10+ years of software development (Skyscanner, Skype, JP Morgan alumni)
● Full-stack, iOS, Android, (Windows Phone)
Introduction
War stories
Trading systems
Oil rig monitoring
XBox One launch
Uber apps rewrites
Payment systems
A war story:
Uber app rewrite
Continuous testing at scale
Continuous testing at scale
Testing is hard
Iterating is part of the journey
Why do we test?
We test to ship no bugs.
Bug-free code of substance
But at what cost?
Why do we test?
To minimize the business impact of mistakes
while maintaining good execution speed.
We will cover testing of
mobile apps.
Still, a lot of the concepts apply
across the stack.
Crashes
Functional
Bugs
UI Bugs
We test, so we can avoid:
… at Scale … at Uber … Tools & a
Framework
Continuous testing...
… at Scale … at Uber … Tools & a
Framework
Continuous testing...
Initial Team
Team Growth
● 600+ cities, 65+ countries, 6 continents
● 10 engineering offices (4x US, Amsterdam, Denmark, 2x India, Sofia,
Vilnius)
● 18,000+ people, of which 2,500+ engineers & 400+ mobile engineers
Some Uber facts
Hundreds of mobile
engineers?
Request a ride
Fare split
Cash
Uber for Business
Credit card rewards points
Promotions
Promotions
Safety
Over 10 ways to pay
Scheduled
rides
Drive with Uber
Uber Eats, Freight,
Bike, Rental...
Experimentation
65+ countries,
600+ cities
Performance
Cash
Instant payments
Maps & navigation
uberPOOL
Driver incentives
App health
Developer tools
Networking
Feed cards
Driver experience
Driver recognition
Airport pickup
Uber Family
Beacon
Campaigns
Fraud EATS app
Driver app
Freight app
Restaurants app
Other apps
Fleet app
What can “at scale” mean?
● More functionality
● More users & regions, locales
● More code
● More engineers
● More engineering offices & locations
● More automated testing
● More apps
● More functionality
● More users & regions, locales
● More code
● More engineers
● More engineering offices & locations
● More automated testing
● More apps
What does “at scale” mean?
● More bugs
● Smaller/local bugs have bigger impact
● Longer build times
● Communication overhead
● Developer systems need to work 24/7
● Longer time to run tests
● The same problems repeating
Problems
What does “at scale” mean?
… at Scale … at Uber … Framework
Continuous testing...
A few things I found different @Uber compared to my previous experience:
● No formal QA role, testing teams or dedicated DevOps team
● Dedicated team(s) owning testing infrastructure & developer tooling
● More formal planning process
● No staging systems: test tenancies instead
● Blameless postmortem culture
Engineering culture
Continuous testing process @Uber
Write code & land
to master
Pre-release
testing
Ship to users
Continuous testing process @Uber
Write code & land
to master
Pre-release
testing
Ship to users
Continuous
Integration
arc diff
Phabricator diff
Local
validations
Code
reviewers
● Commit message validation
(e.g. test plan, revert plan)
● Linting
Herald
rules
Rules like:
● “If certain files are touched, add
{certain people} as reviewers
● If the files added contain a certain
phrase, add a comment to the diff
Build results
Do a build with:
● Linting
● Unit tests
● Static code analysis
Create a pull request
Herald rules
Herald rule example
● Our lint rules are extensive, evolved since the early years
● NEAL: our language agonistic linting platform (open sourced)
Linting: a first class citizen
Continuous
Integration
arc diff
Phabricator diff
Local
validations
Lint,
Build,
Test
Update the diff
arc
land
“Merge to master”
Code Repo
Submit
Queue
Do a “full” build with:
● Linting
● Unit tests
● Static code analysis
● UI testsBuild Result
Validation
pass
Build speeds matter (even) more, as the team grows
Continuous testing process @Uber
Write code & land
to master
Pre-release
testing
Ship to users
Continuous testing process @Uber
Write code & land
to master
Pre-release
testing
Ship to users
● Local checks (linting)
● Continuous Integration (linting,
unit tests, static analysis)
● Code review
● Safe merging to master (UI
tests, SubmitQueue)
Continuous testing process @Uber
Write code & land
to master
Pre-release
testing
Ship to users
Ready for production
release.
Merge code to master
Release
candidate ?
master
Build cut
Automated tests
Manual tests
Manual testing (sanity)
Manual testing (sanity)
Test tenancy
Staging Production
code (master)
Test
accounts
Production
accounts
Production
accounts
Test
accounts
Test
tenancy
Production
tenancy
Staged rollout
code (master)
Staging & production systems Production system with test tenancy
Ready for production
release.
Merge code to master
Release
candidate
master
Build cut
Automated tests
Manual tests
Dogfooding
bugreports
Dogfooding
Dogfooding: sending bug reports
Bug reporter tool
Phabricator
ticket
Take
screenshot
Teams triage
Ready for production
release.
Merge code to master
Release
candidate
master
Build cut
Automated tests
Manual tests
Dogfooding
bugreports
Crash reports
Ready for production
release.
Merge code to master
Release
candidate
master
Build cut
Automated tests
Manual tests
Dogfooding
bugreports
Crash reports
Localization
...
Fix
Hotfix
Build Train
Continuous testing process @Uber
Write code & land
to master
Pre-release
testing
Ship to users
● Manual testing (sanity)
● Dogfooding
● Crash reports
● Build train
Continuous testing process @Uber
Write code & land
to master
Pre-release
testing
Ship to users
Facts
● Bugs will be introduced that none of the previous tests catch
● With native apps
○ New builds can take days to ship due to the app store approval
process
○ Users might not update their apps for a while.
Conclusion
● Every change should be revertable, remotely.
● Let’s use backend-controlled feature flags
Rolling out to production on mobile
Remote Bugfixing: Feature Flags
Rollout can be risky if the population is large & there is no monitoring.
Staged rollout
● Control user exposure in early stages via a feature flag
● Monitor the impact on key business metrics at each stage
Rolling out to production (not just) on mobile
Ready for production
release.
Staged rollout
Monitor
Rolled out
Rolling out a new feature
Staged rollout monitoring for business impact: statistically significant differences
Monitoring: business events
Monitoring: performance
Continuous testing at scale
Continuous testing process @Uber
Write code & land
to master
Pre-release
testing
Ship to users
● Staged rollout
● Monitoring & alerting
○ Crash reports
○ Business events
○ Performance
The mobile testing lifecycle
Write code & land
to master
Pre-release
testing
Ship to users
In production
Build cut Release
Staged rollout
& monitoring
Code & functional quality
checks
Functional & UX quality
checks, hotfixes
Are we done testing?
Rolled out
Things will catch fire
The mobile testing lifecycle
Write code & land
to master
Pre-release
testing
Ship to users
In production
Build cut Release
Staged rollout
& monitoring
Code & functional quality
checks
Functional & UX quality
checks, hotfixes
Uh-oh...
Monitor & triage issues/alerts
The mobile testing lifecycle
Write code & land
to master
Pre-release
testing
Ship to users
In production
Build cut Release
Staged rollout
& monitoring
Code & functional quality
checks
Functional & UX quality
checks
Outages
Uh-oh...
Monitor & triage issues/alerts
How can we make sure this
does not happen again?
Blameless postmortems
The goal of a postmortem
Understand the root cause in order to take
action to prevent the same issue from impacting
customers again.
The 5 whys
The mobile testing lifecycle
Write code & land
to master
Pre-release
testing
Ship to users
In production
Requirements &
planning
Product & engineering spec, with testing plan
Outages &
postmortems
Uh-oh...
“We did not do proper planning.”
“We did not test this edge case.”
“We did not have a test plan.”
The mobile testing lifecycle @Uber
Write code & land
to master
Pre-release
testing
Ship to users
In production
Requirements &
planning
Staged rollout
& monitoring
Code level quality checks Functional & UX quality
checks
Outages &
postmortems
Monitor & triage issues/alerts
Spec & testing plan
Build cut Release
Rolled out
… at Scale … at Uber … Tools & a
Framework
Continuous testing...
What worked for us, will
not (exactly) work for you.
Why do we test?
To minimize the business impact of mistakes
while maintaining good execution speed.
Continuous testing: tools
Crashes
Functional
Bugs
UI Bugs
We test, so we can avoid:A few tools to detect / avoid:
Continuous testing toolset
Crashes
Functional
Bugs
UI Bugs
● Crash reports
● Crash report
alerting
● Code reviews
● Unit testing
● UI testing
● Manual testing
● Dogfooding
● Staged rollout
● Manual testing
● Dogfooding
● Screenshot testing
A few tools to detect / avoid:
Continuous testing toolset
Crashes
Functional
Bugs
UI Bugs
A few tools to detect / avoid:
● Crash reports
● Crash report
alerting
● Code reviews
● Unit testing
● UI testing
● Manual testing
● Dogfooding
● Staged rollout
● Manual testing
● Dogfooding
● Screenshot testing
Other things
impacting
the business
● Business monitoring
& alerting
● Performance testing
/ monitoring
● (Tools that might
work for you)
Continuous testing toolset
Crashes
Functional
Bugs
UI Bugs
A few off the shelf tools to detect / avoid:
● Crash
reporting:
Crashlytics
● Code reviews
○ Github
○ In-house: Phab
● CI
○ Travis CI / Bitrise
○ In-house: Jenkins
● Manual testing:
crowdsourced platforms
● Screenshot testing
● UI testing
○ XCTest
○ Espresso
Other things
impacting
the business
● Analytics: GA, Mixpanel
● In-house analytics:
Kafka, Elastisearch &
Grafana + ML
● Performance testing
○ XCode & Android
studio profilers
A framework to think about testing
The Continuous Testing Pyramid
Manual
tests
UI tests
Unit Tests
Dog
fooding
Blameless
postmortems
Code reviews
Continuous
integration
Monitor
Alert
Triage
Things going wrong
for customers
Team owning testing infrastructure
To make all of this scale:
Improve
processes
& systems
All engineers
All engineers
All engineers
All teams
All employees
All teams
Continuous testing at Scale
Why do we test?
To minimize the business impact of mistakes
while maintaining good execution speed.
As you scale, iterate on the tools you use, your team
structure & processes to keep doing this.
Gergely Orosz
Engineering Manager, Uber Amsterdam
Thank you Open sourced tools for more efficient testing
● uber.github.io
● Language agonistic linting platform: NEAL
● Android
○ Nanoscope (tracing tool)
○ NullAway (static checks to avoid
NullPointer exceptions)
○ OkBuck: use the buck build system on
a gradle project
@GergelyOrosz
eng.uber.com
Proprietary and confidential © 2018 Uber Technologies, Inc. All rights reserved. No part of this
document may be reproduced or utilized in any form or by any means, electronic or mechanical,
including photocopying, recording, or by any information storage or retrieval systems, without
permission in writing from Uber. This document is intended only for the use of the individual or entity
to whom it is addressed and contains information that is privileged, confidential or otherwise exempt
from disclosure under applicable law. All recipients of this document are notified that the information
contained herein includes proprietary and confidential information of Uber, and recipient may not
make use of, disseminate, or in any way disclose this document or any of the enclosed information
to any person other than employees of addressee to the extent necessary for consultations with
authorized personnel of Uber.

More Related Content

Similar to Continuous testing at scale (20)

PDF
Introduction to-automated-testing
BestBrains
 
PPTX
Uber Mobility Meetup: Mobile Testing
Apple Chow
 
PDF
UPC Plone Testing Talk
Timo Stollenwerk
 
PDF
Continuous Testing Improve Efficiency and Ship Better Software.pdf
Steve Wortham
 
PDF
Agile Testing Pasadena JUG Aug2009
Grig Gheorghiu
 
PDF
Software Testing
Andrew Wang
 
PDF
Automated testing-whitepaper
imdurgesh
 
PPTX
CI/CD for mobile at HERE
Stefan Verhoeff
 
PPTX
Testing 101
Noam Barkai
 
PDF
How To Fit Testing Into The Iteration
Rally Software
 
PDF
Test Driven Development With YUI Test (Ajax Experience 2008)
Nicholas Zakas
 
PDF
Mobil Weekend - Evolution of the Test Team
Csaba Szabó
 
PDF
Testing Without Waste - Automatic Testing
Futurice
 
PPTX
Testing
thehoagie
 
PDF
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...
Roberto Pérez Alcolea
 
PDF
Getting your mobile test automation process in place - using Cucumber and Cal...
Niels Frydenholm
 
PDF
Pair programming pair testing working together with the developers by Simon ...
Agile ME
 
PPTX
Testing Best Practices
Axway Appcelerator
 
PDF
Continuous testing in agile projects 2015
Fabricio Epaminondas
 
PDF
Agile Tools for Mobile
Kevin Rohling
 
Introduction to-automated-testing
BestBrains
 
Uber Mobility Meetup: Mobile Testing
Apple Chow
 
UPC Plone Testing Talk
Timo Stollenwerk
 
Continuous Testing Improve Efficiency and Ship Better Software.pdf
Steve Wortham
 
Agile Testing Pasadena JUG Aug2009
Grig Gheorghiu
 
Software Testing
Andrew Wang
 
Automated testing-whitepaper
imdurgesh
 
CI/CD for mobile at HERE
Stefan Verhoeff
 
Testing 101
Noam Barkai
 
How To Fit Testing Into The Iteration
Rally Software
 
Test Driven Development With YUI Test (Ajax Experience 2008)
Nicholas Zakas
 
Mobil Weekend - Evolution of the Test Team
Csaba Szabó
 
Testing Without Waste - Automatic Testing
Futurice
 
Testing
thehoagie
 
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...
Roberto Pérez Alcolea
 
Getting your mobile test automation process in place - using Cucumber and Cal...
Niels Frydenholm
 
Pair programming pair testing working together with the developers by Simon ...
Agile ME
 
Testing Best Practices
Axway Appcelerator
 
Continuous testing in agile projects 2015
Fabricio Epaminondas
 
Agile Tools for Mobile
Kevin Rohling
 

More from Gergely Orosz (6)

PPTX
Payments Integration at Uber: a (Short) Case Study
Gergely Orosz
 
PPTX
Mobile Architecture at Scale
Gergely Orosz
 
PPTX
Success on the Marketplace, App Store and Apps Marketplace
Gergely Orosz
 
PPTX
Wp7 performance challenges
Gergely Orosz
 
PPTX
Developing for Windows Phone 7
Gergely Orosz
 
PPT
An Introduction To Silverlight
Gergely Orosz
 
Payments Integration at Uber: a (Short) Case Study
Gergely Orosz
 
Mobile Architecture at Scale
Gergely Orosz
 
Success on the Marketplace, App Store and Apps Marketplace
Gergely Orosz
 
Wp7 performance challenges
Gergely Orosz
 
Developing for Windows Phone 7
Gergely Orosz
 
An Introduction To Silverlight
Gergely Orosz
 
Ad

Recently uploaded (20)

PDF
CIFDAQ Weekly Market Wrap for 11th July 2025
CIFDAQ
 
PDF
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
PDF
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
PDF
Chris Elwell Woburn, MA - Passionate About IT Innovation
Chris Elwell Woburn, MA
 
PDF
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
PDF
From Code to Challenge: Crafting Skill-Based Games That Engage and Reward
aiyshauae
 
PPTX
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PDF
Python basic programing language for automation
DanialHabibi2
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
PDF
SWEBOK Guide and Software Services Engineering Education
Hironori Washizaki
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PDF
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
How Startups Are Growing Faster with App Developers in Australia.pdf
India App Developer
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
PDF
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
PDF
July Patch Tuesday
Ivanti
 
CIFDAQ Weekly Market Wrap for 11th July 2025
CIFDAQ
 
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
Chris Elwell Woburn, MA - Passionate About IT Innovation
Chris Elwell Woburn, MA
 
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
From Code to Challenge: Crafting Skill-Based Games That Engage and Reward
aiyshauae
 
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
Python basic programing language for automation
DanialHabibi2
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
SWEBOK Guide and Software Services Engineering Education
Hironori Washizaki
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
How Startups Are Growing Faster with App Developers in Australia.pdf
India App Developer
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
July Patch Tuesday
Ivanti
 
Ad

Continuous testing at scale

  • 1. Continuous Testing at Scale Gergely Orosz, Engineering Manager @GergelyOrosz 8 May 2018
  • 2. ● Engineering manager @Uber, in Amsterdam ● 10+ years of software development (Skyscanner, Skype, JP Morgan alumni) ● Full-stack, iOS, Android, (Windows Phone) Introduction
  • 3. War stories Trading systems Oil rig monitoring XBox One launch Uber apps rewrites Payment systems
  • 4. A war story: Uber app rewrite
  • 8. Iterating is part of the journey
  • 9. Why do we test? We test to ship no bugs.
  • 10. Bug-free code of substance But at what cost?
  • 11. Why do we test? To minimize the business impact of mistakes while maintaining good execution speed.
  • 12. We will cover testing of mobile apps. Still, a lot of the concepts apply across the stack.
  • 14. … at Scale … at Uber … Tools & a Framework Continuous testing...
  • 15. … at Scale … at Uber … Tools & a Framework Continuous testing...
  • 18. ● 600+ cities, 65+ countries, 6 continents ● 10 engineering offices (4x US, Amsterdam, Denmark, 2x India, Sofia, Vilnius) ● 18,000+ people, of which 2,500+ engineers & 400+ mobile engineers Some Uber facts
  • 19. Hundreds of mobile engineers? Request a ride Fare split Cash Uber for Business Credit card rewards points Promotions Promotions Safety Over 10 ways to pay Scheduled rides Drive with Uber Uber Eats, Freight, Bike, Rental... Experimentation 65+ countries, 600+ cities Performance Cash Instant payments Maps & navigation uberPOOL Driver incentives App health Developer tools Networking Feed cards Driver experience Driver recognition Airport pickup Uber Family Beacon Campaigns Fraud EATS app Driver app Freight app Restaurants app Other apps Fleet app
  • 20. What can “at scale” mean? ● More functionality ● More users & regions, locales ● More code ● More engineers ● More engineering offices & locations ● More automated testing ● More apps
  • 21. ● More functionality ● More users & regions, locales ● More code ● More engineers ● More engineering offices & locations ● More automated testing ● More apps What does “at scale” mean? ● More bugs ● Smaller/local bugs have bigger impact ● Longer build times ● Communication overhead ● Developer systems need to work 24/7 ● Longer time to run tests ● The same problems repeating Problems
  • 22. What does “at scale” mean?
  • 23. … at Scale … at Uber … Framework Continuous testing...
  • 24. A few things I found different @Uber compared to my previous experience: ● No formal QA role, testing teams or dedicated DevOps team ● Dedicated team(s) owning testing infrastructure & developer tooling ● More formal planning process ● No staging systems: test tenancies instead ● Blameless postmortem culture Engineering culture
  • 25. Continuous testing process @Uber Write code & land to master Pre-release testing Ship to users
  • 26. Continuous testing process @Uber Write code & land to master Pre-release testing Ship to users
  • 27. Continuous Integration arc diff Phabricator diff Local validations Code reviewers ● Commit message validation (e.g. test plan, revert plan) ● Linting Herald rules Rules like: ● “If certain files are touched, add {certain people} as reviewers ● If the files added contain a certain phrase, add a comment to the diff Build results Do a build with: ● Linting ● Unit tests ● Static code analysis Create a pull request
  • 30. ● Our lint rules are extensive, evolved since the early years ● NEAL: our language agonistic linting platform (open sourced) Linting: a first class citizen
  • 31. Continuous Integration arc diff Phabricator diff Local validations Lint, Build, Test Update the diff arc land “Merge to master” Code Repo Submit Queue Do a “full” build with: ● Linting ● Unit tests ● Static code analysis ● UI testsBuild Result Validation pass
  • 32. Build speeds matter (even) more, as the team grows
  • 33. Continuous testing process @Uber Write code & land to master Pre-release testing Ship to users
  • 34. Continuous testing process @Uber Write code & land to master Pre-release testing Ship to users ● Local checks (linting) ● Continuous Integration (linting, unit tests, static analysis) ● Code review ● Safe merging to master (UI tests, SubmitQueue)
  • 35. Continuous testing process @Uber Write code & land to master Pre-release testing Ship to users
  • 36. Ready for production release. Merge code to master Release candidate ? master Build cut Automated tests Manual tests
  • 39. Test tenancy Staging Production code (master) Test accounts Production accounts Production accounts Test accounts Test tenancy Production tenancy Staged rollout code (master) Staging & production systems Production system with test tenancy
  • 40. Ready for production release. Merge code to master Release candidate master Build cut Automated tests Manual tests Dogfooding bugreports
  • 42. Dogfooding: sending bug reports Bug reporter tool Phabricator ticket Take screenshot Teams triage
  • 43. Ready for production release. Merge code to master Release candidate master Build cut Automated tests Manual tests Dogfooding bugreports Crash reports
  • 44. Ready for production release. Merge code to master Release candidate master Build cut Automated tests Manual tests Dogfooding bugreports Crash reports Localization ... Fix Hotfix
  • 46. Continuous testing process @Uber Write code & land to master Pre-release testing Ship to users ● Manual testing (sanity) ● Dogfooding ● Crash reports ● Build train
  • 47. Continuous testing process @Uber Write code & land to master Pre-release testing Ship to users
  • 48. Facts ● Bugs will be introduced that none of the previous tests catch ● With native apps ○ New builds can take days to ship due to the app store approval process ○ Users might not update their apps for a while. Conclusion ● Every change should be revertable, remotely. ● Let’s use backend-controlled feature flags Rolling out to production on mobile
  • 50. Rollout can be risky if the population is large & there is no monitoring. Staged rollout ● Control user exposure in early stages via a feature flag ● Monitor the impact on key business metrics at each stage Rolling out to production (not just) on mobile
  • 51. Ready for production release. Staged rollout Monitor Rolled out Rolling out a new feature
  • 52. Staged rollout monitoring for business impact: statistically significant differences
  • 56. Continuous testing process @Uber Write code & land to master Pre-release testing Ship to users ● Staged rollout ● Monitoring & alerting ○ Crash reports ○ Business events ○ Performance
  • 57. The mobile testing lifecycle Write code & land to master Pre-release testing Ship to users In production Build cut Release Staged rollout & monitoring Code & functional quality checks Functional & UX quality checks, hotfixes Are we done testing? Rolled out
  • 59. The mobile testing lifecycle Write code & land to master Pre-release testing Ship to users In production Build cut Release Staged rollout & monitoring Code & functional quality checks Functional & UX quality checks, hotfixes Uh-oh... Monitor & triage issues/alerts
  • 60. The mobile testing lifecycle Write code & land to master Pre-release testing Ship to users In production Build cut Release Staged rollout & monitoring Code & functional quality checks Functional & UX quality checks Outages Uh-oh... Monitor & triage issues/alerts
  • 61. How can we make sure this does not happen again?
  • 63. The goal of a postmortem Understand the root cause in order to take action to prevent the same issue from impacting customers again.
  • 65. The mobile testing lifecycle Write code & land to master Pre-release testing Ship to users In production Requirements & planning Product & engineering spec, with testing plan Outages & postmortems Uh-oh... “We did not do proper planning.” “We did not test this edge case.” “We did not have a test plan.”
  • 66. The mobile testing lifecycle @Uber Write code & land to master Pre-release testing Ship to users In production Requirements & planning Staged rollout & monitoring Code level quality checks Functional & UX quality checks Outages & postmortems Monitor & triage issues/alerts Spec & testing plan Build cut Release Rolled out
  • 67. … at Scale … at Uber … Tools & a Framework Continuous testing...
  • 68. What worked for us, will not (exactly) work for you.
  • 69. Why do we test? To minimize the business impact of mistakes while maintaining good execution speed.
  • 70. Continuous testing: tools Crashes Functional Bugs UI Bugs We test, so we can avoid:A few tools to detect / avoid:
  • 71. Continuous testing toolset Crashes Functional Bugs UI Bugs ● Crash reports ● Crash report alerting ● Code reviews ● Unit testing ● UI testing ● Manual testing ● Dogfooding ● Staged rollout ● Manual testing ● Dogfooding ● Screenshot testing A few tools to detect / avoid:
  • 72. Continuous testing toolset Crashes Functional Bugs UI Bugs A few tools to detect / avoid: ● Crash reports ● Crash report alerting ● Code reviews ● Unit testing ● UI testing ● Manual testing ● Dogfooding ● Staged rollout ● Manual testing ● Dogfooding ● Screenshot testing Other things impacting the business ● Business monitoring & alerting ● Performance testing / monitoring ● (Tools that might work for you)
  • 73. Continuous testing toolset Crashes Functional Bugs UI Bugs A few off the shelf tools to detect / avoid: ● Crash reporting: Crashlytics ● Code reviews ○ Github ○ In-house: Phab ● CI ○ Travis CI / Bitrise ○ In-house: Jenkins ● Manual testing: crowdsourced platforms ● Screenshot testing ● UI testing ○ XCTest ○ Espresso Other things impacting the business ● Analytics: GA, Mixpanel ● In-house analytics: Kafka, Elastisearch & Grafana + ML ● Performance testing ○ XCode & Android studio profilers
  • 74. A framework to think about testing
  • 75. The Continuous Testing Pyramid Manual tests UI tests Unit Tests Dog fooding Blameless postmortems Code reviews Continuous integration Monitor Alert Triage Things going wrong for customers Team owning testing infrastructure To make all of this scale: Improve processes & systems All engineers All engineers All engineers All teams All employees All teams
  • 76. Continuous testing at Scale Why do we test? To minimize the business impact of mistakes while maintaining good execution speed. As you scale, iterate on the tools you use, your team structure & processes to keep doing this.
  • 77. Gergely Orosz Engineering Manager, Uber Amsterdam Thank you Open sourced tools for more efficient testing ● uber.github.io ● Language agonistic linting platform: NEAL ● Android ○ Nanoscope (tracing tool) ○ NullAway (static checks to avoid NullPointer exceptions) ○ OkBuck: use the buck build system on a gradle project @GergelyOrosz eng.uber.com
  • 78. Proprietary and confidential © 2018 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.