Drive Faster Quality Insights through Customized Test Automation
0 likes•221 views
The document discusses the importance of customized test automation for improving quality insights and release velocity in software development. It highlights key challenges faced during quality analysis, the desired outcomes for quality feedback, and introduces the Digital Zoom solution that facilitates effective communication and reporting across teams. The document also emphasizes the need for structured quality dashboards and a tagging strategy to enhance visibility and collaboration throughout the software development lifecycle.
Drive Faster Quality Insights through Customized Test Automation
- 1. Drive Faster Quality Insights through Customized Test Automation
- 2. Presenters Tzvika Shahaf Director Product Management Perfecto Eran Kinsbruner Dir. Technical Evangelist Perfecto
- 3. Fast Feedback and Quality Analysis Drives Better Release Velocity #Of Tests per cycle #Number of Executions #Release Frequency
- 4. Current Mobile Market State
- 5. Versions: N, N-1, Beta Windows OS: 7, 8.1, 10 Mac OS X: Sierra, El-Capitan 15 Combinations Versions: N, N-1, Beta Windows OS: 7, 8.1, 10 9 Combinations Windows 7, 8.1, 10: IE 11 Windows 7: IE 8, 9 5 Combinations Versions: 10.1, 9.1.3 Mac OS X Sierra: 10.1 Mac OS X El-Capitan, Yosemite: 9.1.3 3 Combinations Windows 10: Edge 141 Combinations 33 Combinations For Max Market Coverage * Validate against your web traffic analytics Current Web Market State
- 6. Did You Ever Ran Into Such Conversation? 4/24/2017 6© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 7. Customer Research Shows How Much Time Is Spent On Test Automation Regression Analysis Post Execution? • 1 Hour – 10% • 2-4 Hours – 14% • 3-5 Hours – 18% • 1 Business Day – 20% • I Don’t Know – 30% 4/24/2017 7© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 8. Quality Analysis Key Challenges 4/24/2017 8© 2017, Perfecto Mobile Ltd. All Rights Reserved. Test Data Analysis Time Is too long Fit Analysis Into Existing Processes and Tools For quality decision making Limited Quality Health visibility across platforms 1 2 3
- 9. Desired State of Quality Feedback Across the SDLC On-Demand access to digital quality dashboard and custom reports Ability to drive fast Go/No Go decision based on quality trends Bridge gaps and drive ongoing communication across Dev, QA, and Biz Focus more on feature development and validation, less on analysis Empower continuous quality while eliminating flaky tests for faster cycles ENHANCED QUALITY VISIBILITY THROUGHOUT THE PIPELINE MEET THE FEEDBACK WINDOW FOR RELEASE (VELOCITY GOALS) CAN I RELEASE A NEW PATCH TO PROD IN LIKE 3 HOURS? Cross Team Tools Alignment For Zero Blind Spots Optimized quality strategy across the entire organization Test Data Analysis Time Is too long Fit Analysis Into Existing Processes and Tools For quality decision making Limited Quality Health visibility across platforms 1 2 3
- 11. Digital Zoom™ - Using Reports Across The SDLC 4/24/2017 11© 2016, Perfecto Mobile Ltd. All Rights Reserved. Developer QA Manager/ExecutivesDev & Test Engineer Agile/Feature Teams • Show me my build status • Point me to the exact breaking point • I’d like to get data using Jira, Jenkins and Slack • Cross feature team communication through Slack, Jira and custom reports • DevOps quality visibility per feature team and post integration phase • Have the quality visibility to support decisions during the SDLC • Understand where problems are in order to distribute across right people/teams • Understand trends and challenges over time • Help me optimize quality using tags and custom fields • Maximize context during test authoring • Work within my IDE/FW
- 12. Introducing RTDD (Reporting Test Driven Design) RTDD Is a method that enables teams to design quality with the debugging and insight in mind (context based). Among the key benefits of RTDD are 1) Structured quality dashboard that can be sliced and diced by various persona’s 2) Fast MTTR of defects unrelated to the test suite size 3) Reliable and much more stable test suite (less flaky tests) 4) Low risk Go/No-Go decision due to a data-driven dashboard and quality artifacts 4/24/2017 12© 2015, Perfecto Mobile Ltd. All Rights Reserved. • Structured test authoring via SDK • testStart() • testStep() • testStop() • Context based Tagging (“regression”)
- 13. Benefits of Tagging Strategy 4/24/2017 13© 2016, Perfecto Mobile Ltd. All Rights Reserved. Logical Steps (incl. BDD support) Tags Heat Map Based on Tags Fast Quality Analysis/On-Demand Status Context and Tags Drive Distribution for Analysis For quality drill down and triaging process
- 14. Drilling Down Capabilities • Quality dashboard in the digital era means: • Cross platform visibility vs. single platform reports • Customizable report with API’s vs. locked with off the shelf solutions • Any dev language • Any IDE • Any test framework • Data driven risk management based on digital platform data • 3 layer reporting • Single test report (STR) • Suite level report (Grid) • Complete Dashboard View (By category, By platform, By tags) 4/24/2017 14© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 15. Defect Triaging – DigitalZoom™ Artifacts For RCA • Digital reporting artifacts supports fast MTTR • Logs • Videos • Screenshots • Stack Trace • Vitals • Network PCAP files • PDF report • Export report Using Public API (Json) for further processing 4/24/2017 15© 2015, Perfecto Mobile Ltd. All Rights Reserved. Video Stack Trace Screenshots (incl. expected vs. Actual)
- 16. CI Dashboard Enables Build Trending And Overall Quality History Visibility 4/24/2017 16© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 17. CI Dashboard Enables Build Trending And Overall Quality History Visibility 4/24/2017 17© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 18. Communication & Alignment Empowers Quality Collaboration 4/24/2017 18© 2015, Perfecto Mobile Ltd. All Rights Reserved. Seamless Integrations
- 19. LIVE DEMO
- 20. Thank You