Webinar issues we_find_slideshare
Download as PPTX, PDF•0 likes•355 views
The document discusses performance testing best practices across various environments, including lab, staging, and production, highlighting goals, benefits, and limitations at each stage. It emphasizes the importance of identifying issues related to code, hardware, and network performance, particularly as the scale of testing increases. Several case studies illustrate real-world challenges faced by companies and the solutions implemented to enhance application performance.
Webinar issues we_find_slideshare
- 1. Best Practices in Performance Testing Issues Uncovered as Scale Increases Webinar Series
- 2. The Test Lab Goals: • Isolate bugs in code • Optimize page performance • Plug memory leaks • Tune databases • Establish performance expectations Limitations: • Production-quality hardware • IT support and resources • Bandwidth and network infrastructure •3rd party application components • Time
- 3. The Staging Environment Goals: • Verify capacity • Establish configurations • Stress available resources • Verify performance expectations Limitations: • Availability / scheduling • Not full-scale • Excludes key components • Time
- 4. The Full Production Environment Goals: • Test to and beyond expected limits • Verify entire infrastructure • Confirm load balancing and firewall integrity • Certify performance in normal and extreme conditions Benefits: • All components • 3rd party apps • All integrations • Continuous • Global latencies • Reality
- 5. Performance isn’t One-Dimensional “You have a fever.” Now what? External web load User Experience QA OPS
- 6. Performance is Multi-Dimensional Find problems fast when metrics are viewed in context, at scale External and Internal user load Complex Web Applications : AJAX, HTML5, video, mobile, web services User Experience Multi-Dimensional Views
- 7. Performance is Multi-Dimensional Find problems fast when metrics are viewed in context, at scale External and Internal user load Complex Web Applications : AJAX, HTML5, video, mobile, web services Application Deployment Deployment Component & Infrastructure Infrastructure Network User Experience Code (Internal) (External) Extract Transport Load Real-Time End to End The CloudTest Performance Data platform is built for scalability and real- In-Memory time identification OLAP of issues Multi-Dimensional Views
- 8. Extending Performance Beyond the Lab Discovering issues across teams, time and scale Stage / Team Debugging code Garbage collection Memory leaks Memory leaks Database thread counts Dev & Test Slow pages Inefficient database queries Method-level tuning Test Lab (a few thousand users) Scale of Test
- 9. Extending Performance Beyond the Lab Discovering issues across teams, time and scale Stage / Team Configuration Setting & Sizing Auto-scaling failures Release Max sockets exceeded Global latency variance & Deploy Slow third-party plug-ins Security bottlenecks Default configuration settings Search technology limits Inadequate server resources Garbage collection Memory leaks Memory leaks Database thread counts Dev & Test Slow pages Inefficient database queries Method-level tuning Test Lab Staging (thousands to 10’s of thousands) Scale of Test
- 10. Extending Performance Beyond the Lab Discovering issues across teams, time and scale Tuning for full-scale…and well beyond Latency between systems Network Network configuration & Ops Network bandwidth CDN file placement Conflict with other apps DNS routing Stage / Team Load Balancer configuration Firewall max capacity Auto-scaling failures Unbalanced web servers Release & Deploy Max sockets exceeded s Global latency variance Security bottlenecks Slow third-party plug-ins Default configuration settings Search technology limits Inadequate server resources Garbage collection Memory leaks Memory leaks Database thread counts Dev & Test Slow pages Inefficient database queries Method-level tuning Test Lab Staging Production (100% +++) Scale of Test
- 11. CloudTest Live Real-world Issue Resolution Company Profile Project Profile Company Toy Manufacturer Environment Lab and Staging Size Fortune 500 Application Drupal (Apache, PHP, MySQL) Location Private Hosted Cloud Objective Identify Breaking Points Project Length Five Days (non-consecutive) Lab Staging
- 12. CloudTest Live Real-world Issue Resolution • First test cycle in the lab • Some servers hitting 100% CPU Lab • 239 hits per second • Bottleneck in throughput • 40.5 MBit/second bandwidth
- 13. CloudTest Live Real-world Issue Resolution • Second test cycle in the lab • Some servers hitting 100% CPU Lab • Higher concurrency target • Performance problem over time • Bandwidth stayed the same • Load balancer flapping servers
- 14. CloudTest Live Real-world Issue Resolution • Duration problem was crippling – could not recover Lab
- 15. CloudTest Live Real-world Issue Resolution • Bandwidth – ouch! Staging • CPU at 100%
- 16. CloudTest Live Real-world Issue Resolution • 767 Home pages served at 650 megs total transfer. • That’s an 850kb page weight! Yikes! Staging
- 17. CloudTest Live Real-world Issue Resolution Mission Debriefing - Fortune 500 Toy Manufacturer Lab Staging • Poor throughput on the initial tests • Massive page weight on the home page • CPU hits 100% • Application unresponsive at ~2000 users • Bandwidth cap identified in the lab • Database connections leaking over time • Load balancer dropping servers Outcome • Customer happy with the breaking point identified • Optimized web assets in the home page • Offloaded assets to the CDN • Load balancer algorithm optimizations
- 18. CloudTest Live Real-world Issue Resolution Company Profile Project Profile Company Media and Broadcasting Environment Production Size Fortune 500 Application Java Location Private Data Center Objective 25,000 Concurrent Project Length Three Days Production
- 19. CloudTest Live Real-world Issue Resolution • Steadily increasing response times • Constant stream of errors (< 1%) Production • Variety of error types
- 20. CloudTest Live Real-world Issue Resolution • One rogue page – streaming the TV episode Production
- 21. CloudTest Live Real-world Issue Resolution • Connection reset errors starting the video stream • Entitlement license server calls failing Production
- 22. CloudTest Live Real-world Issue Resolution Mission Debriefing - Fortune 500 Media Company Production Outcome • Connection issues with CDN • Worked with CDN provider to reinforce certain geographies • One problem page • Added capacity to the license server cluster • Massive bandwidth usage
- 23. CloudTest Live Real-world Issue Resolution Company Profile Project Profile Company Advanced Technology Environment Production Size Fortune 50 Application Java (Struts), Oracle Multiple HostedPrivate Location Data Centers Objective 172,000 Concurrent Project Length Two Months Production
- 24. CloudTest Live Real-world Issue Resolution • Recurring error pattern during scenario completion Production
- 25. CloudTest Live Real-world Issue Resolution Awesome outcome in Production!
- 26. CloudTest Live Real-world Issue Resolution Mission Debriefing - Fortune 50 Technology Company Production Outcome • Load balancer firmware bug with 2048 bit cert • Worked with hardware LB manufacturer on a fix • Database issues with the login process • Really low error rate • Successful launch!
- 27. Q&A Please join us for the next webinar in this series: Testing Mobile Applications May 3rd at 10AM PDT, 1PM EDT Register at: http://bit.ly/ezq3fP Contact SOASTA: www.soasta.com/cloudtest/ [email protected] 866.344.8766 Follow us: twitter.com/cloudtest facebook.com/cloudtest
Editor's Notes
- #2: Good Morning and welcome to the second in our new series, Best Practices in Performance Testing. This series is designed to share with you the best practices our performance engineering team is developing as they work with world-class customers of all sizes. Today we will be exploring what issues are uncovered as customers test applications starting a small scale in development and moving to full-scale testing in production. My name is Betsy Zikakis, I’m the VP, Marketing for Soasta and I’ll be your MC today.
- #28: Thank you so much for attending this webinar. The archive will be available on our website by the end of this week. Please tell others about it. Our next webinar in this series is focused on testing mobile applications. You can go to the link on this slide to register now. It will be up on the SOASTA home page to register tomorrow morning. And you will receive an email invitation next week.