Build a Large Scale In-House Test Lab for Mobile Apps
Download as PPTX, PDF•3 likes•4,028 views
The document discusses building an in-house test lab for mobile app testing. It outlines the benefits of continuous testing on real devices versus outsourcing. It also covers challenges like device fragmentation and how tools like Testdroid can help automate testing. Infrastructure needs like device control servers, USB hubs and WiFi are reviewed. Tips provided include having an authentic testing environment and integrating testing into the development cycle.
Build a Large Scale In-House Test Lab for Mobile Apps
- 1. Building a Large-Scale In-House Test Lab for Mobile Apps 4 June 2014 Ville-Veikko Helppi Technical Product Manager [email protected]
- 2. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 2 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
- 3. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 3 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
- 4. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 4 Manual vs. Automation Smaller coverage, More money burnt & time wasted, Error- prone Large coverage, quickly completed, Less money & time wasted, Exact results.
- 5. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 5 Prerequisites • Testing environment as authentic as possible – Real devices <-> emulators/simulators • Test coverage needs to be adequate – Software is not identical in all Android/iOS devices – Hardware is even more fragmented landscape • Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes
- 6. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 6 In-House Mobile App Testing • Continuous Testing with real devices • Cloud/Out-of-Premises vs. In-House • Improve app ratings & your brand reputation • Operational costs vs. Lost revenue? • Consequences of not-being-tested-enough?
- 7. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 7 Reasons to Build In-House Labs • Pre-Market devices or other hardware • Corporate security policies • Availability and usage level • Special setups
- 8. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 8 Core Benefits of CI and Testing • Version control and build revision • Build automation, instant execution & testing • All types of testing & integrations enabled • Frequent commits, Code consolidation -> Fast builds • CI/Testing is the best fit for mobile agile process
- 9. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 9 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
- 10. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 10 Why Apps Fail?
- 11. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 11 EXAMPLE: Android devices Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
- 12. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 12 EXAMPLE: Android OEMs Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
- 13. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 13 EXAMPLE: Android OS Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/
- 14. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 14 OS Versions ICS JB KK 17.7M test runs, 288 unique Android devices, 50+ configurations
- 15. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 15 Display Resolution 17.7M test runs, 288 unique Android devices, 50+ configurations
- 16. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 16 Memory
- 17. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 17 Coverage? IS THAT ALL YOU NEED? Picture by Double Encore - http://www.doubleencore.com/2014/05/android-user-data-2014/
- 18. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 18 Data, results & setup information were gather between 1 January – 31 March 2014 (Q1’2014) - 17.7M test/device runs - 288 unique devices - Thousands of userswww.testdroid.com Research
- 19. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 19 Global Coverage
- 20. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 20 Robust/Unstable – Q1’14 The Most Robust Devices Samsung Infuse 4G SGH-I997 Lenovo A60 Dell Streak 7 M02M001 HTC ChaCha A810e Coolpad 7295 Samsung Galaxy Europa GT-I5500 Samsung Acclaim SCH-R880 Motorola Motosmart XT389 Sharp Aquos Phone Zeta SH-02E Samsung Galaxy mini GT-S5570 The Most Unstable Devices Samsung Galaxy S IV Active SGH-I537 Sony Xperia J ST26i Samsung Galaxy S IV Altius SCH-R970 Samsung Galaxy S IV Altius SGH-M919 Samsung Galaxy Note 8.0 GT-N5110 Samsung Galaxy S IV Altius SGH-I337 Samsung Galaxy Mega GT-I9205 HTC One M7 4.3 Asus Transformer Pad TF300TL Sony Xperia Acro S LT26w 99.805 % 99.797% 99.794% 99.785% 99.783% 99.780% 99.778% 99.775% 99.771% 99.770% 29.722% 36.376% 39.007% 41.455% 43.284% 44.737% 45.536% 45.794% 47.689% 50.576 %
- 21. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 21 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
- 22. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 22 Hardware – Top 3 Things • Device Control Servers – Hard disks – RAM – Energy Efficiency • USB Hubs • WiFi Infrastructure
- 23. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 23 Software – Top 3 Things • Testdroid Enterprise • Continuous Integration – e.g. Jenkins CI • Other agile tools
- 24. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 24 How Testdroid Helps? • Speed up dev & testing cycles significantly • Agile development & integration with mobile practices • Continuous Testing
- 25. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 25 Simple Setup Device clusters Device Control Server(s) Cluster #1 Cluster #2 Cluster #3 USB Hubs, Cables Cluster #4 Cluster #5
- 26. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 26 Developers Source code Repository Build failed Successful build QA & Testers Error reporting Test Cases Application Commit Users / Consumers
- 27. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 27 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
- 28. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 28 1
- 29. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 29 2
- 30. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 30 3
- 31. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 31 4
- 32. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 32 5
- 33. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 33 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
- 34. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 34 Testdroid Update
- 35. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 35 Testdroid Update
- 36. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 36 Stay tuned! A Step-by-step guide how to build an in- house test lab for mobile apps. Will be available at www.testdroid.com Coming Soon!
- 37. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 37 Deployment Options
- 38. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 38 Only $999! For Trial Purpose! We’re giving a 30-day trial license with Mac Mini. In worst case, you got a Mac Mini with $999 – and it the best scenario you’ll have a product that will save you a lot of money, time and results as a better application!
- 39. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 39 Agenda • Intro & Benefits: In-House Mobile App Testing • Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software • Tips & Tricks for Building In-House Lab • Testdroid Update • Q&A Agenda
- 40. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 40