How to Get Automatic Analysis for Load Test Results
Download as PPTX, PDF•1 like•408 views
The document discusses the importance of automating load test result analysis to improve efficiency in performance testing practices. Key elements include breaking down test results into components, utilizing automation techniques, and distinguishing between 'automated' and 'automatic' processes. It emphasizes correct test signals, thresholds, and the benefits of real-time and post-test analysis using modern tools and techniques.
How to Get Automatic Analysis for Load Test Results
- 1. Automatic Analysis for Load Test Results by Andrey Pokhilko Chief Scientist, BlazeMeter Inc. February 3rd, 2016 Mountain View, CA
- 2. About Me My key interest: tools that help people enjoy their job Current: Chief Scientist for BlazeMeter Past: Performance Engineering in Yandex JMeter Plugins project founder http://jmeter-plugins.org/ Loadosophia author http://loadosophia.org/ Taurus Tool lead dev http://gettaurus.org/
- 3. Look Out of the Window 1. It’s the era of total speed-up 2. Which is solved by total automation 3. Which leads to CI/CD as standard practices You can’t have a human in the loop anymore
- 4. Understand how humans do it, then automate the process
- 5. Analysis Raw Data Information DecisionsFacts True Analysis is Here
- 6. What is our “Load Test Results Analysis” ? 1. Decide if results are reliable 2. Break complex test timeline into components - surprisingly important 3. Decide if it had hit bottleneck, had memory leak etc 4. Understand APM correlations 5. Decide if something has changed since the last test / over recent N tests
- 7. When do we do it? 1. Online while test is still running 2. Post-test 3. Comparing tests to a specific baseline 4. Observing KPI history from a series of tests Unconsciously most of the time Speed Constraint main challenge Resources Used
- 8. Challenge Accepted Isn’t it trivial? Wasn’t it done before? Why can’t we reuse what APM vendors do these days?
- 9. How can we do it?
- 11. Apply the right test signal First and foremost:
- 12. Right test signal Load Profile Real test intention Measured ResponseSystem
- 13. Common Test Signals aka Load Profiles Stress, Max Throughput => Capacity, “how much” Constant, Soak, Sustain => Quality, “how good” Spike, recovery Combinations => reality requires to adapt Stress Constant Mixed Recovery
- 14. Each testing process stage has own techniques
- 15. In-Test Automation Instant threshold Windowed threshold => average / instant Steadiness threshold => stop constant test earlier Decisions to be made: stop test, mark it failed, mark it passed
- 16. In-Test Automation: Implementations JMeter-Plugins Autostop - show it Taurus Pass/Fail criteria - show it BlazeMeter Thresholds
- 17. Post-Test Analysis Stage breakdown 1. Ramp-Up Stage 2. Constant Stage 3. Leftovers POI detection APM correlation - “measure everything” costs you in the analysis stage
- 18. Post-Test Analysis: Implementations BlazeMeter App => Thresholds Loadosophia.org: stages breakdown test classification refined KPIs - it’s all about speed monitoring correlation 85% to 95% correct
- 19. Test Comparison & Series Trend Have reference test (aka “baseline”) Raise flag on too much difference Notify on trend outlier Notify on trend reaching limit It’s all easy to automate with APIs + simple scripts
- 20. We’ve Learned Today 1. Why to automate results analysis 2. What our “result analysis” is 3. Prerequisites to make it automatable 4. Particular techniques & implementations
Editor's Notes
- #4: Automated analysis is very important because of CI/CD era and today's agile practices and things like microservices. It all gives you tens of services to test, tens of times per day. You physically can't analyse it by human eye, you need automation.
- #6: Give example of JMeter: JTL file with samples => graphs => facts seen from graphs => decision if build is ok to go
- #8: ‘cause moments we do it put important restrictions and requirements
- #9: Today, most of the load testing tools and services do not do analysis for load test results, they do presentation. Automating this is not easy, because the process of a load test is complex and produces a timeline that needs to be analyzed.
- #12: To solve that we need first to set the rules of the load test to make the results as predictable as possible. Many people do not think of their test intentions.
- #13: The Model
- #17: Even if you stopped ramping-up test - so what? Next thing is to tell KPIs from stress-test. Some autostops allow you to remember beginning point of problems
- #18: Then, we need to get the automatic analysis method that is able to work with timeline data and teach it to recognize patterns that we used to recognize with our eyes. Which conclusions to make automatically Many people do not think of their test intentions. => how it detects intentions Technical analysis like forex, although it’s not to predict, but to deduce some facts Which KPIs to take 85% to 95% user satisfaction unsatisfied used to improve the algo Monitoring correlation allows you to go back to level of resources to understand what limits you