Mobile Device Client Application Performance Testing
12 likes6,245 views
The document outlines the first part of a mobile performance testing webinar series focusing on testing client applications on mobile devices. It discusses the importance of mobile performance testing, types of testing, and a case study analyzing the performance of the game Angry Birds on various Android devices. Key strategies for performance testing include defining requirements, selecting devices, and utilizing specific tools to evaluate performance against predetermined criteria.
Mobile Device Client Application Performance Testing
- 1. Mobile Performance Testing Part 1: How to Performance Test Mobile Client Applications on a Mobile Device
- 2. Series of 3 Mobile Performance Webinars Part 1: Clients applications – Now Part 2: The Server – Feb 21 Part 3: The Network – March 14 All at 10 AM EST
- 3. Alan Trefzger Newman Yang Jan Princen
- 4. Software Quality Improvement • XBOSoft – Software QA Consulting and Testing – Currently 100+ QA experts – San Francisco, Beijing, Oslo, Amsterdam
- 6. “Proven Results” “Our products are very complex. We expected a one year ramp up time, but XBO was able to contribute to our release cycles in just 6 months." - Development Vice President, Hyperion-Oracle "The results are very encouraging. Thanks a lot for all the effort put into this project. We really appreciate your work." - Manager, Quality Assurance, Autodesk "Everything you've done is impressive and we are quite pleased, well done! So keep up the good work." - QA Director, CA
- 7. Agenda • Importance of Mobile Performance Testing • Types of Mobile Performance Testing • Mobile Performance Testing for Local (Android) Applications – Setting up a performance test – Using different tools to test • Case study – How fast can Angry Birds run • Evaluate the tools • Q&A
- 8. Importance of Performance • Mobile internet traffic expected to pass desktop internet traffic in 2014. • By 2015, over half of all mobile subscribers are expected to be engaged in m-payments . • Amazon reported that the company calculated it lost 1% of sales for every extra tenth of a second required to load a page. • Google experiment: traffic and revenue fell by 20% when the pages with more results took an extra half second to load. • Akamai found that the number of users who abandon a page after 3 seconds was 57%. - Mobile Site Optimization, Sep 2011, Strangeloop
- 9. Types of Mobile Performance 1. Client Application / device performance 2. Server performance 3. Network performance
- 10. Server Performance Following are some of the challenges, variables, and issues covered in the Server Performance webinar. • Amount of data sent, • Number of communications between Server and Client/Browser • Tools • Optimization Strategies
- 11. Network Performance Following are some of the challenges or variables covered in the Network Performance webinar. • Latency, jitter, packet loss,… • 2.5G, 3G, 4G LTE • CDNs
- 12. Local Device and App Performance Why focus on the local device and App? • Discover performance based on different hardware/software configurations with the same application for a defined task. • Discover which platform meets a set performance objective with the least cost. • Determine which platform uses the least resources.
- 13. Performance is important to almost every application or task Buy a product Obtain a bank balance Look up a ticker symbol Kill a man with Download one page spear (WoW) of information Start an application (game)
- 14. How Fast can Angry Birds Run?
- 15. Performance Test Strategy Performance test on a few devices, and extrapolate that it will work on those devices that are more powerful (more memory, higher CPU Hz, etc…, 5 criteria are defined later). Different than functional test. It may work functionally on a certain device, but not meet performance requirements. Example, for Android, you can test on a well chosen set of 7-15 devices and be confident that your application will work functionally on the majority of Android devices. However some of those devices may not meet performance requirements.
- 16. Defining the test Product Manager wants to find the lowest phone that satisfies the performance requirements. • Requirements: – Launch Angry Birds in 15 seconds – Throw an Angry Bird far away in 8 seconds – Play Angry Birds and play music at the same time – Play Angry Birds and at the same time delete 100 SMS’s in 5 seconds Lowest solution that fits requirements Using tools and manual testing to make purchasing decision
- 17. Steps for performance test 1. Define task, requirements and test. 2. Choose a sample set of devices. 3. Choose one or more tools and run them. 4. Compare to manual test. 5. Use Quadrants’ DB of devices to eliminate those phones which do not meet performance requirements. 6. Chose a mobile device based on criteria other than performance.
- 18. Configuration of devices • 3 popular Android devices
- 19. Tools • Quadrant – measures CPU, Memory, I/O, 2D and 3D. • Linpack - measures CPU floating point performance by single thread and multi thread. • Smartbench – Simple tool to get idea if the device is suited for example gaming. • Others: Jbenchmark, SPMark, Nenamark
- 20. Quadrant Result Total Score averages the scores of the 5 specific criteria 3500 3000 2500 2000 Lenovo A1 1500 HTC Desire HD 1000 Motorola 500 ME860 0 Total Score CPU Memory I/O 2D 3D
- 22. Using Linpack 60 50 40 Lenovo A1 30 HTC Desire HD 20 Motorola ME 860 10 0 Single Thread Multi Thread MFLOPS - Millions of Floating Point Operations per Second Lenovo A1 HTC Desire HD Motorola ME 860 Single Thread 9.91 39.01 38.94 (MFLOPS) Multi Thread 8.55 32.51 51.96 (MFLOPS)
- 23. Using Smartbench 3000 2500 2000 Lenovo A1 1500 HTC Desire HD 1000 Motorola ME860 500 0 Productivity Game Motorola ME860 >HTC Desire HD > Lenovo A1
- 24. Manual Test Angry Bird
- 25. Manual Test Result - Angry Birds Scenarios Required Lenovo A1 HTC Desire HD Motorola ME 860 1. Launch Angry Bird 15 seconds 19 seconds 14 seconds 10 seconds V2.1.1 2. Throw the bird to a 8 seconds 8 seconds 6 seconds 5 seconds far distance 3. Play Angry Bird Slight V2.1.1 and play music at Continuous Continuous Continuous discontinuity the same time 4. Play Angry Birds and delete 100 SMS 5 seconds 4 seconds 3 seconds 2 seconds simultaneously
- 26. Results of Manual test • Lenovo A1 - didn’t meet requirements • HTC Desire HD - met requirements • Motorola ME 860 - met requirements Conclusion • Since the Lenovo A1 device didn’t meet the performance requirements, any device with a lesser Quadrant score than the Lenovo will be rejected. • Any device with an higher score than the HTC device will be accepted.
- 27. Tying it all together 1. Define task, requirements and test – Angry Birds, certain speed, multi apps 2. Choose a sample set of devices – Lenovo, HTC, Motorola 3. Choose a tool (0ne or more) – Quadrant, (Linpack, Smartbench) 4. Compare to manual test – Find out which device conform to performance requirements 5. Use Quadrants’ DB of devices to eliminate those phones which do not meet performance requirements. 6. Chose a mobile device based on criteria other than performance.
- 28. Thanks Questions and Answers www.xbosoft.com [email protected] 408-350-0508 28