Don’t Be Another Statistic! Develop a Long-Term Test Automation Strategy
- 1. T21 Test Automation 5/5/16 15:00 Don't Be Another Statistic! Develop a Long-term Test Automation Strategy Presented by: Lee Barnes Utopia Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com/
- 2. Lee Barnes Utopia Founder and CTO of Utopia Solutions Lee Barnes is responsible for the firm's delivery of software quality solutions which include process improvement, performance management, test automation, and mobile quality. Lee has more than twenty years of experience in the software quality assurance and testing field. He has successfully implemented test automation and performance testing solutions in hundreds of environments across a wide array of industries. A recognized leader in his field, Lee speaks regularly on related topics.
- 3. © Utopia Solutions Don’t Be Another Statistic! Develop a Strategy for Long-term Test Automation Success Lee Barnes, CTO Utopia Solutions
- 4. © Utopia Solutions Agenda •What is the end goal of test automation? •Why do we need a strategy? •How do we know if we’re ready to succeed? •What should a strategy look like? Questions to Answer Today…
- 5. © Utopia Solutions SECTION 1 What is Our End Goal?
- 6. © Utopia Solutions4 Common Test Automation Goals Test Coverage Resource Value Test Cycle Time
- 7. © Utopia Solutions5 Test Automation Success Criteria • Issue detection and recovery • Accurate verification • Unattended execution Reliable • Minimum sensitivity to application and test case changes • Test case definition separate from automation code Maintainable • Test coverage expanded efficiently • Automated test cases created by non-automation resources Scalable
- 8. © Utopia Solutions SECTION 2 Why Do We Need a Strategy?
- 9. © Utopia Solutions7 Why Test Automation Fails Organizational • Lack of support • Unrealistic expectations • Disjointed org structure • Missing skill sets Process • Immature testing process • Segregated processes • Lack of test environment / data control Technical • Poor tool support for AUT • Under or over design automation approach
- 10. © Utopia Solutions SECTION 3 How Do We Know if Were Ready?
- 11. © Utopia Solutions9 Are We Ready?
- 12. © Utopia Solutions10 Test Automation Readiness Assessment Org • Structure • Support • Goals and Expectations • Resources and Skills Process • Test Maturity • Test Assets • Process Integration • Test Environment / Data Tech • AUT Technology • AUT / Tool Integration Assessment Areas
- 13. © Utopia Solutions Organizational Questions • Expectations – What are our goals for test automation? – Do we understand the effort and cost to implement and maintain test automation – Do we understand the required skill sets for successful test automation? • Structure – How is the testing function organized? – Where are the automation resources relative to everyone else? • Resources – Are our automation engineers allowed to focus on automation? – Do our automation engineers have the appropriate skills?
- 14. © Utopia Solutions12 Common Assessment Findings – Organization Distributed Testing / Automation Efforts App App App App • Uneven automation resource demand • Inconsistent approach • Duplicated efforts • Restricted skill growth and knowledge sharing Leads to…
- 15. © Utopia Solutions Process Questions • Test Process – What are the testing activities that can benefit from automation? – Do we have testing assets that can promote automation? • Development Process – What type of development process is used for the AUT? – Are the development and testing processes integrated? • Test Environment – What environments do we have available for test automation? – Do we have control over the test data in these environments?
- 16. © Utopia Solutions14 Common Assessment Findings – Process Analyze Design Build Test Deploy No ability to identify impact of app changes on automated test suite Unexpected / Unpredictable Maintenance
- 17. © Utopia Solutions15 Process Analyze Design Build Test Deploy Capture potential impacts as early as possible to enable proactive maintenance Capture App Changes Analyze Impact Maintain Execute Automation Activities Application SDLC
- 18. © Utopia Solutions16 Common Assessment Findings - Environment Shared Test Environment Uncontrolled Test Data Integration Functional Performance Integration Test • Schedule conflicts • Unreliable execution • Unnecessary “hardening” of the test suite Leads to… Automation
- 19. © Utopia Solutions Technical Questions • Applications Under Test – Do we know which AUTs should be targeted for test automation? – Do we know the technologies of the AUTs in scope for test automation? – Where are the AUTs in their application lifecycle? • Test Automation Tool – Do we have access to an automation tool (or tools)? – Does our automation tool appropriately support our AUTs? – If we don’t have a tool, do we understand how to select one?
- 20. © Utopia Solutions SECTION 5 What Does a Strategy Look Like?
- 21. © Utopia Solutions19 Mobile Test Strategy Topics Test Automation Strategy Process & Organization Environment TechnicalResources Scope & Roadmap
- 22. © Utopia Solutions Ideal Scope Attributes Testing Support Activity • Environment / data setup • Tedious output verification Test Coverage • Focused on high frequency test testing activity • NOT focused on automating manual test scripts verbatim • Test cases distilled into levels AUTs • Supported completely by test tool • Multiple AUTs from same development framework / technology • Frequent release cycles • Large regression effort
- 23. © Utopia Solutions Ideal Organizational Attributes Structure • Centralized test automation function • Automation roles are defined Goals and Expectations • Automation is an investment • 100% automation is not likely • Automation requires specialized skills • Automation will free up resources to work on high- value tasks Support • Executive management has visibility to automation • Automation has a dedicated budget • Automation is represented appropriately on project plans Resources and Skills • Resources are dedicated to automation • Resources are skilled in automation as a process AND technology
- 24. © Utopia Solutions Ideal Process and Environment Attributes Test Process • Test objectives are clear and measurable • Test cases are defined and documented • Expected results are predictable • Basic metrics for testing and automation SDLC Integration • AUT in a frequent release state with visibility to future roadmap • Visibility to system changes as early as possible in release cycle Test Environment • Test environment dedicated to automation • Test data controlled
- 25. © Utopia Solutions23 Automation Strategy – Roadmap • Begin with required organizational and process changes • Overlap with base framework and minimal coverage • Enhance and add test coverage over small iterations Create Implementation Timeline Phase nPhase 1 Continuous Process & Organizational Improvements Base Framework Initial Coverage AUT 1 Framework Enhancements Expand Coverage Initial Coverage Phase 2 AUT 2 Expand Coverage Expand Coverage Initial Coverage Phase 3 AUT 3
- 26. © Utopia Solutions SECTION 3 Miscellaneous Thoughts…
- 27. © Utopia Solutions25 Test Results Test Automation Approach Test Automation Framework Automated Test Cases Test Definition Test Execution Non-technical / Test Analysts & Subject Matter Experts Technical / Automation Engineers
- 28. © Utopia Solutions26 Regression Test Automation and Agile Cycles App Dev Cycle (Sprints) User Stories Application Automation Requirements Automation Dev Cycle Automated Test Suite Development and automation cycles NOT necessarily in sync
- 29. © Utopia Solutions27 Key Summary Points • Test automation is NOT a tool! • Understand your readiness to succeed from these perspectives… • Organization • Process & Environment • Technical • Develop a strategy to address readiness gaps and implement in small phases
- 30. © Utopia Solutions28 Questions… …. and Answers! Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: [email protected] Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com