uTest STPcon spring 2012 presentation
Download as PPTX, PDF•0 likes•856 views
The rise of social, local, and mobile technologies is causing a revolution in how software is developed and tested. Testing teams must now test applications across a fragmented landscape of devices, locations, and social integrations. This new environment presents challenges for replicating real-world conditions and comprehensively testing applications without visibility into third-party services that are constantly changing. Testing teams need to develop new strategies to keep pace with these trends and ensure their opinions are represented when priorities are set.
uTest STPcon spring 2012 presentation
- 1. The SoLoMo Revolution And The Resulting Tectonic Shift In Testing | Matt Johnston | @matjohnston | CMO @ uTest | March 2012
- 2. The Challenge SoLoMo Huh? | 2
- 3. SoLoMo Breakdown So Lo Mo Social Local Mobile | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 3
- 4. SoLoMo Coming Fast In the past six months… | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 4
- 5. QA Leaders Better Sit Down • Media & analysts aren’t the only ones talking about it – Marketing – Product – Engineering – IT – Sales – C-suite • Want a seat at the table? – Get informed – Get involved – Develop an opinion – Advocate strongly | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 5
- 6. The Challenge Trends In Social Apps | 6
- 7. Not Just For B2C Anymore • The social revolution isn’t coming… it’s here – Nothing is disconnected – No industry is immune • The stats on social media adoption in Fortune 500 firms: – 77% integrate Twitter – 84% incorporate LinkedIn – 93% utilize Facebook – 57% publish on YouTube • Gartner: By 2016, social integrated w/ nearly all apps – Websites & apps – Mobile apps & mobile web – Desktop clients | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 7
- 8. Goes Soooo Much Further | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 8
- 9. The Challenge Trends In Local Apps | 9
- 10. Not Just For Marketing • Businesses use location for more than just marketing • Growing trend toward location-based intelligence in apps – Asset tracking – Maps & directions – Location finders – Check-in services – Geo-based personalization – Location-based ads | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 10
- 11. Increasing Granularity Good news for users (and marketers)… good luck to testers. | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 11
- 12. The Challenge Trends In Mobile Apps | 12
- 13. Mobile Apps Boom Market • ABI: mobile revenues will hit $133 billion in 2014 • B2B mobility no longer just for email – Business apps: CRM, ERP, HR systems – Productivity apps: docs, spreadsheets, presentations – Collaboration apps: email, IM, publishing – Medical apps: health records, patient education • B2C mobility growth even steeper – Retail: location-intelligent m-commerce – Media: magazines & newspapers going purely digital – Travel: mobile bookings, check-ins, maps, deals – Education: tablets in every classroom – Healthcare: patient records, physician notes – Social: no explanation required – Gaming: ditto | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 13
- 14. Mobile’s Just Getting Started | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 14
- 15. Did I Mention… | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 15
- 16. Psst… It’s Not Just Mobile • Smartphones & tablets make QA immeasurably tougher • Now imagine a whole world of apps: E-Readers Cars Household Appliances Smart Homes Gaming Consoles Connected TVs | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 16
- 17. The Challenge SoLoMo Impact On Testing | 17
- 18. Brave New World • Impact on design, dev & IT well-recognized & resourced • Sadly, but not at all surprisingly, not true for testing – Want proof? Specialized designers & devs… but who tests? | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 18
- 19. Social: Impact On Testing • Social presents functional test challenges – Testing code & integrations that aren’t yours… – That constantly changes… – And you get the blame • As well as security & privacy pitfalls – Internally, a channel for spam, malware and data breaches – Confidential data may be posted and trigger regulatory penalties – Externally, increasingly used for user authentication on your products | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 19
- 20. Social: Impact On Testing • New challenges include how will you: – Regression test multiple social integrations for web & mobile apps? – Write (and execute) test cases with zero visibility into social’s roadmaps? – Automate anything? – Test sharing/liking in a QA environment without publishing via social? – Test your social footprint? - FB page (and apps) - LinkedIn page - Twitter page – Ensure security/privacy if you enable users to use social sign-on? | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 20
- 21. Local: Impact On Testing • Not just a mobile problem – Localization testing – Geo-based personalization • But it is a mobile problem too – Apps used outside the confines of the QA lab, under in-the-wild conditions • Impossible to test effectively from only inside the lab | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 21
- 22. Local: Impact On Testing • New challenges include, how will you: – Replicate location (answer: you don’t) – Test in NYC, SF, LA, DFW, CHI? – Test abroad - SIM cards - Carriers – Test GPS-based intelligence? – Test IP-based localization? | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 22
- 23. Mobile: Impact On Testing • New challenges include how will you: – Native apps: Test across ever-changing OS, devices and carriers? – Mobile web: Test across wide range of OS and mobile browsers? – Either way, how does your testing replicate real-world conditions - Specific and varied locales - Tower-to-tower hopping - Roaming - Low battery - Touch screen actions (pinch, swipe, pan, etc) - Portrait vs. landscape - Display definition | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 23
- 24. Mobile: Impact On Testing • Mobile web vs. native apps – 2009: 5-horse race – 2010: 4-horse race – 2011: 2-horse race (depending upon your industry) – 2012: 3 horses (including HTML5-driven mobile web) | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 24
- 25. F|R|A|G|M|E|N|T|E|D • Android alone (and that’s only by OS version) | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 25
- 26. Mobile Testing Landscape • Form factors matter again – Phones (feature & smart) – Tablets – Hybrids • Prohibitive HW & SW costs • Here we go again – Late-90s web, part deux - Lack of mature tools - Lack of usability standards - Lack of security standards - Extremely fluid landscape | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 26
- 27. Multiple Launch Schedules Multiple products, roadmaps & launch schedules | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 27
- 28. Test Coverage Web Testing Matrix Is Tough Features OS & Browser | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 28
- 29. Test Coverage Mobile Testing Matrix Is Insane Features Handset Makers OS & Browser & Models | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 29
- 30. Test Coverage Mobile Testing Matrix Is Insane Features Wireless Handset Makers Carriers OS & Browser & Models | 30 In-The-Wild Testing for Functional + Security + Load + Usability + Localization
- 31. Test Coverage Mobile Testing Matrix Is Insane Features Location Wireless Handset Makers Carriers OS & Browser & Models | 31 In-The-Wild Testing for Functional + Security + Load + Usability + Localization
- 32. Increasingly Complex Testing • The laughably simplistic notion of a “test matrix” | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 32
- 33. The Challenge Ummmm… So What Do We Do? | 33
- 34. Sufficiently Freaked Out?!? • Don’t be… the good news is that testing ain’t dead! – Chance for QA leaders to step up… – To lead the charge… – To defend users and app quality | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 34
- 35. …And The Bad News… • You should be scared… need to completely re-think testing – Can’t automate our way out of this – Can’t hire or train our way out – Can’t solve this problem from soley within the lab • More distributed problem than ever before – Need a new testing mindset – Need to bridge the growing gap - Between testing & users - Between artificial and real - Between in-the-lab & in-the-wild testing | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 35
- 36. Lab-Based Testing Innovation Manual vs. Automation Managing vs. Leading Teams In-House vs. Outsourced Agile vs. Waterfall Exploratory vs. Testers vs. Test Cases Engineers Offshore vs. Nearshore | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 36
- 37. Users Work & Live In The Wild I n The Wi l In-The-Wild Testing for Functional + Security + Load + Usability + Localization | 37
- 38. To Make Matters Worse • Users are less tolerant of quality failures • Social media & app stores give everyone a megaphone • And their problem… is your problem | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 38
- 39. The Challenge A Call To Action For Test Leaders (Two, Actually) | 39
- 40. 1. Look Beyond The Lab • Yes, keep testing in-the lab – Keep training in-house – Keep outsourcing – Keep automating – Keep innovating • Recognize it will never again be sufficient on its own • Build your company’s plan for testing in-the-wild – Real-world conditions: Live testers, real devices, imperfect connectivity – Mirror your user base - Technologically: OS, browser, anti-virus, device, carrier - Geographically: Continent, country, city, language - Demographically: Age, gender, education, employment, industry, hobby • In-house QA leadership is vitally important – You have the business and product context – You know what your firm cares about – You need to be the orchestrator who leads this symphony - What to test in the wild, how to test in the wild, holistic test planning | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 40
- 41. 2. Be The Voice For QA • SoLoMo is coming – with or without QA’s buy-in • Want QA to be heard? Want quality to be built in? – Don’t wait: No one else is coming to the rescue – Get informed: Develop an thoughtful opinion – Get involved: Find out when/where decisions are being made – Be proactive: Build a plan for quality from the ground up – Be protective: You’re a protector of the user… of the brand | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 41
- 42. The Challenge SoLoMoWoSto A Few Stories (if there’s time) | 42
- 43. The Challenge Questions? Answers. Matt Johnston | CMO @ uTest [email protected] | @matjohnston | 43