Small is Beautiful- Fully Automate your Test Case Design
Download as PPTX, PDF1 like780 views
This webinar will demonstrate how to fully automate testing and test case design using Grid Tools' Agile Designer software. The software can automatically generate the smallest number of test cases needed to maximize coverage from requirements. It can then push these test cases out as automated tests, removing bottlenecks caused by manually creating, executing, and maintaining tests. Attendees will learn how to automatically generate optimized test cases and scripts from models of their system under test and keep tests up to date as requirements change.
Small is Beautiful- Fully Automate your Test Case Design
- 1. WEBINAR Small is Beautiful – How to Fully Automate your Testing and Test Case Design Huw Price, Managing Director – Grid Tools
- 2. • Automatically generate the smallest number of test cases from requirements for maximum coverage • Push test cases out as automated tests • Remove bottlenecks caused by the manual creation, execution and maintenance of test cases This Agile Designer™ webinar will demonstrate how you can:
- 3. Huw Price Managing Director and co-founder of Grid-Tools
- 4. What’s Supposed to Happen TesterBusiness AnalystUser Developer The end user knows what they want The analyst specifies what that is. The programmer writes the code The tester tests the program
- 5. What’s going wrong? • Requirements are stored disparately & In different formats • They are typically incomplete and ambiguous Requirements Code Test Cases Test Data Environments The developer requires clarification & defects enter the code • Test cases are manually created and executed • Provide just 10-20% coverage and defects are detected late • Cannot respond to change requests • Never get deleted • Contains sensitive information, and time is spent securing it • Provides poor coverage, and testers spend half their time finding or making data • Is large, and 20% of the SDLC is spent finding data • Is unavailable ‘upstream’ Change Request • Are shared with other teams or are not complete yet • Are legacy or expensive mainframe systems Hardware & Data Dependencies • Automation is used late • Scripts are manually developed • It cannot respond to change • Tests the same functionality Automation
- 6. Grid Tools - Products • Automation Scripts • User Stories • Optimized Test Cases • Backlog and Complexity • Test Data • Requirements and Coverage • Data Coverage • Find and Allocate Test Data • Subset and Mask Data • Synthetic Data • Virtualized Messages • Test Data on Demand • Selenium • QTP • IBM Rational • Ranorex • SilkTest • Microsoft UI Automation • CA • Parasoft Javelin Automation Dataviz Javelin Data Orchestration
- 7. A Simple Form Easy to test?
- 8. Test case design • Currently manual • No formal processes. • “Exploratory Testing” the norm: poor requirements lead to poor overall testing, with testers having to fill in the gaps • No linkage to test data – process is manual, painstaking and very time-consuming. • No flexibility for change requests: a critical weakness in an agile environment. Changes take longer than the original requirement!
- 9. What is Testing Coverage? The Business Thinks it’s: • Code Covered • Number of Test Covered – Tests Run • Percentage of use cases • All Paired Combinations It is actually: • Designing Sufficient Tests To VERIFY That The Design And Code Correctly Implement The Requirements • Did you get the right answer for the right reason - Two or more defects may sometimes cancel each other out - Observability
- 10. Different Coverage Techniques Combinatorial – All Pairs – Constrained All Pairs • Does not support Expected Results - You have to work it out for each combination which is very time consuming • Combinatorial is something that ‘accidentally’ increases functional coverage (and only to a point) • Combinatorial does not give you any solid metrics on how good your testing actually is – you can only infer that x out of y combinations have been covered, but it has no bearing in actual functional logic. • You end up with lots of false errors that have to be checked by hand
- 11. Copyright Bender RBT Inc. 2009 11RBT24261 Cause-Effect Graphing Observable Events and Path Sensitizing • Assume C and F are not observable events. • Assume A is stuck at FALSE. • Enter as a test case A(T), B(T), D(T), E(T). • Results should be C(T), F(T) and G(T). T T T T T T T Different Coverage Techniques
- 12. Different Coverage Techniques Optimized Flow Chart Modelling • Advanced Graph Optimization • Most projects already have a flow chart • Similar results to Cause and effect • Supports Looping • Supports Constraints • Not applicable for non sequential based logic
- 13. Optimizer Demo • Different Optimization Methods • Managing Change
- 14. The path explorer tool and optimization algorithms Create Perfect Test Cases • Generate the smallest number of test cases with maximum coverage • Test more functionality in fewer tests • Measure test coverage and know that every requirement has been tested The cost and complexity tool in Agile Designer™
- 15. Model Based Testing – Step by Step
- 16. Model Based Testing – Step by Step CUSTOMER NUMBER: 123456 CUSTOMER NAME: HUW PRICE CUSTOMER COUNTRY: UK VALIDATION PROCESS VALIDATION PROCESSDATA UPDATED
- 17. Validation Coverage - Add Related Data Logic ALL POSSIBLE ALL PAIRS ALL EDGES
- 18. Model Based Testing – Process Coverage ALL POSSIBLE ALL PAIRS ALL EDGES & PATH HINTS
- 19. Model Based Testing Most testing is – Frankly: • Random • Unstructured • Repetitive • Not thorough enough • Can’t be measured • Can’t keep up • Too slow Model based testing, lets you define what is supposed to happen and then test that. Model based testing is: • Accurate • Structured • Thorough • Measurable • Can keep up with change • Allows you to measure Risk • Lets you automate very quickly
- 20. Automated tests: manual generation • Automated testing frameworks are heavily scripted • Script generation is usually done manually • Alternative solution: use scriptless automation frameworks – But your back to Manual test case design • Optimal solution: from automated test case generation, through automated script generation, to automated execution. Automating the automation.
- 21. Active Automation •If you could create better manual tests faster, would you? •If you could create automation scripts as quickly as manual test cases, would you?
- 22. Active Automation • Scripts Are Created Automatically • Testing is Model Based • Use a component library of covered tests • Test Data is created or found as part of the Automation Active Automation
- 23. Active Automation Record an Automation Script Import it into Agile Designer Design the logic Auto create your automation robots If something changes, auto create new robots! Cl Cl Cl Cl N O H N N N H O Cl Cl Cl Cl
- 24. Model Based Testing – Use Libraries of Test Cases
- 25. Agile Designer – Data driven or Flow Driven LOGIC DATA DRIVEN AUTOMATION AUTOMATION HARNESS DATA 1 DATA 2 DATA 3 DATA 4 FLOW DRIVEN AUTOMATION 1 – DATA 1 AUTOMATION 2 – DATA 2 AUTOMATION 3 – DATA 3 AUTOMATION 4 – DATA 4 AUTOMATION 5 – DATA 5 You can drive using a static flow and feed it with test data and expected results You can drive by auto generating brand new test scripts for each path Or a combination
- 26. Automation Demo • Capture Automation Assets • Define the logic • Create the automation robots
- 27. Shift left the effort of testing Business Analyst User Requirements Programmer Tester Tester Test Cases & Test Scripts Program • Concentrate effort into the requirements gathering stage • User and BA map requirements to unambiguous, active flowcharts • Testers can generate automated test scripts and test cases directly from the requirements • If a user makes a change to the requirements, the tests can be automatically updated
- 28. Active Automation •If you could create better manual tests faster, would you? •If you could create automation scripts as quickly as manual test cases, would you? •Yes please!
- 29. Q & A If you have any further questions please contact us on [email protected]
- 30. For more information on what Agile Designer™ can do for you, please contact: www.agile-designer.com [email protected] or by phone: UK: +44 (0)1865 884 600 or US: +1 888 603 1213 THANK YOU FOR WATCHING
Editor's Notes
- #8: Even with what looks like a very simple requirement there are a large number of potential use cases. In this example the complexity comes from the reset button.
- #15: Point One: generate the perfect set of test cases directly from the requirements From the requirements flowchart, Agile Designer will identify every possible path through the system. Where it gets clever is the path optimization. This uses “deep, dark maths” to identify the smallest number of paths needed to fully test a systems functionality – i.e., to fully test the requirements now that they are fully defined. Users can choose from multiple algorithms, to automatically generate the smallest number of test cases to cover: all possible paths; all edges (arrows in/out of the blocks); all nodes; all in/out edges; all pairs. Automatically generating test cases removes testing bottlenecks: Manually writing test cases and test scripts is slow and error-prone (i.e., it provides poor coverage) For example: 6 hours to produce 11 test cases with 16% coverage (internal) Testing currently takes up around 47% of the SDLC. Path explorer – Look at all possible paths through the functional logic of the flowchart Identify the smallest number of test cases needed to test maximum functionality Store/create your test cases (use cases) Store these test cases – push them out to automation engines, ALM/QC etc. In the path explorer: go to your test cases, and click ‘export special’. Select which folder you want to save them in. Update and add new test cases. Point two: maximum coverage means you can test more for less Agile Designer helps reduce testing time and costs by identifying the smallest number of tests needed. This means that you can systematically improve coverage, knowing that all requirements have been tested. This is in contrast to industry standard, where much testing is redundant, and much functionality goes untested: Over-testing of certain functions by 40 times Typically only 10-20% coverage – negative/unhappy paths go untested Up to 30% of testing time is wasted on duplicate, invalid or redundant tests Examples of optimization: A financial services company created 11 test cases in 6 hours with 16% requirement coverage Agile Designer automatically created 17 test cases in 2 hours with 100% coverage Another project relied on 3 test cases which provided just 5% coverage, this resulted in bugs making it into production which is expensive to fix Agile Designer generated 12 test cases with 100% coverage in 30 minutes In one project, the possible number of cases identified was 326; Agile Designer identified that only 17 were needed for 100% coverage Point Three: The same algorithms used to identify the test cases can be used assess how much functional coverage the stored test cases provide. The notion of perfect test cases is based on the concept of coverage, path modelling and risk-based testing.
- #28: From these test cases…The secret of Agile Designer is that by doing one thing you create multiple outputs, for example by designing the perfect set of test cases you are creating a static test of the requirements document. You can push out Story boards Complexity analysis – how long should this take? Virtualization end points Test Data Automation scripts AND even push actions, use cases and test cases to story boards From a flowchart, you therefore have all the qualitative information needed to work through the development lifecycle, up to the execution of automation scripts. Once you have stored your paths, you simply have to push them back to TMX. From there, you can use them in your automation engine of choice, automatically testing an application iteratively. Typically the BA writes the requirement along with a few diagrams. There are usually lots of words. These have to be interpreted by the programmer and then by the tester. These words (based on each users local domain knowledge and location) are often ambiguous and result in the wrong thing being developed or tested. Even harder are change requests, the amount of work for a simple change request is massive as all existing test cases have to be verified and changed again, and test scripts have to be rewritten. With Agile Designer the hard work is concentrated in the requirements gathering stage – all the hard work is concentrated here.