Test Coverage: An Art and a Science
Download as PPTX, PDF7 likes3,370 views
This document discusses test coverage, which is a measure of how much code is executed by a test suite. It defines test coverage and explains the business impacts. Types of test coverage include statement coverage, branch coverage, path coverage, and condition coverage. The document provides examples and discusses implementation criteria and recognizing problems with test coverage.
Test Coverage: An Art and a Science
- 1. TEST COVERAGE: AN ART AND A SCIENCE WITH JAY PHILIPS AND JEAN ANN HARRISON
- 2. YOUR PRESENTERS Mobile & Software Test Consultant Email: [email protected] Twitter: @JA_Harrison LinkedIn Profile: http://www.linkedin.com/pub/jeanann-harrison/ 4/b55/865/ Company: Project Realms Website: www.projectrealms.com Email: [email protected] Twitter: @Jayphilips LinkedIn Profile: www.linkedin.com/in/jayphilips
- 3. AGENDA • Test Coverage Definition • Business Impacts of Test Coverage • Types of Test Coverage & Example • Implementation • Questions/Answers
- 4. TEST COVERAGE - DEFINITION • A measure of the proportion of a program exercised by a test suite, usually expressed as a percentage. This will typically involve collecting information about which parts of a program are actually executed when running the test suite in order to identify which branches of conditional statements which have been taken.
- 5. BUSINESS IMPACT OF TEST COVERAGE • Results can Assure the quality of test • Results can help identify what portions of the code was actually touched for the release/fix • Results can help identify the paths in your application that weren’t tested • Results can identify all the paths & decision points used in the application, which will allow you to increase test coverage
- 6. TYPES OF TEST COVERAGE
- 8. STATEMENT COVERAGE Statement Coverage testing can be one line of code or a block of code which executes one task. Example might be testing an If statement with no decision but fulfilling the one condition.
- 9. BRANCH COVERAGE The code takes the tester through a series of decisions to be made when the value from the array called age is reached. The decision to do something needs to be tested and the decision to go to the next cast when the value from the array is not the value in the case. Two tests must be done for each case because there are two decisions: if yes – do this and break or no, go to the next case.
- 10. PATH COVERAGE This code takes you down a road of the where the value of a is incremented by 1 after starting at 10. Once incremented by 1, the value of a is printed out with a new line. But, there is a change in the path, when a = 15, the path says to add 1 to the value and then go through the loop. Because a != 15, the value of a is printed. The value of a is increased by 1 and will be printed out until a reaches the value of 20 and the function stops.
- 11. TESTS FOR PATH COVERAGE EXAMPLE The code’s path was written where the value of a equals 11, 12, 13, 14, 16, 17 or 18, 19, that value will be printed as: “Value of a: <numbers listed above>”. When the code’s path reaches 15, the code states that to add 1 to the value of a and then to start the loop over again, never reaching the print statement. The code doesn’t contain further instructions of adding 1 or more to the value of a but instead continues to the code stating to print out the value of a. The tests to follow would include checking the sequence of the values printed as well as what values are printed excluding 15. Is each value represented as “a = <value != 15”. Make sure each printed value starts on a new line. a=11 a=12 a=13 a=14 a=15 a=16 a=17 a=18 a=19
- 12. CONDITION COVERAGE Example of a function written in C++ where test cases are needed to cover the following Boolean conditions where z would = x if: • Test x > 0 but y = 0 • Test x > 0 but y < 0 • Test x < 0 but y > 0 • Test x < 0 but y = 0 • Test x = 0 but y > 0 • Test x < 0 and y < 0 More tests than if x > 0 & y > 0
- 13. TEST COVERAGE FOR TYPES OF TESTS What type of Tests Use in conjunction with: Why implementing Unit Tests debugger tool Can help to manipulate data & simulate error conditions Integration Tests test harness Use a test harness to simulate user interface checking on which components were tested. Regression Tests log files Checking overal application functionality for 2 reasons: 1) anything broken with newly added code 2) checking what previously built code had been tested and touched. Performance Tests log files Security Tests log files Functional Tests log files testing old vs new code – include for all. System Integration Tests log files Production Testing log files
- 14. IMPLEMENTATION CRITERIA Element Consideration (what you said requirement) Criteria Language Support Not all tools support all languages (ex, Java, Cobol) so determine what languages the tool will be used for to determine the tool selected Instrumentation Determine when the instrumentation will be done and if the tool can support it. Ex: Will it be done via instructions or probes? Will be it be via source code or byte code or other? Measurement Determine which measurement type you want to use (branch, block, etc) Reporting Determine what type of reporting you need and ability to distribute Environment/Hardware Determine what environment(s) & hardware are needed to use the tool Training Determine when the tool vendor/consultant can provide training to your teams Setting Expectations Before implementation, a discussion with the management stakeholders of the technical details on the source code: i.e. instrumentation.
- 15. RECOGNIZING AND SOLVING THE PROBLEM
- 16. GETTING STARTED … Creating processes: • Identify users, their purpose for using the software and their industry/context. • Identify external team members (build team, development team, test team etc) • Identify a repository location & setup for completed analysis and level of analysis to be stored. • Identify permissions for repository • Identify recipients of reported analysis & level of detail for the reports • Develop content for reports, create reports
- 17. DOCUMENTING YOUR PROCESS • Documentation Needs (especially for regulated environments) • Setup needs for hardware, storage, team members & their roles • Among team, create & document goals • Chronological flow of tasks • Training tasks
- 18. NEXT STEPS … • Distribute & Reviewing test coverage data to stakeholders • Using the analytics to update test cases for future testing • Rinse and Repeat
- 19. REFERENCES • Slide 4 - Mouse with purple umbrella http://cdn.stylefrizz.com/img/purple-umbrella-mouse.jpg • Slide 5 - Snoopy Happy Dance http://4.bp.blogspot.com/-- u0eQYXh6l8/U15TKvTwSOI/AAAAAAAAEQ0/PQo9QxsvQeE/s1600/pe6 50921_Snoopy+Happy+Dance.jpg • Slide 6 - Code Coverage Text http://en.wikipedia.org/wiki/Code_coverage • Slide 7 – Code Coverage Execution http://www.atollic.com/index.php/trueanalyzer/types-of-code-coverage- analysis • Slide 8 - Statement Coverage http://www.cs.odu.edu/~cs252/Book/stmtcov.html • Slide 9 - Branch code example http://www.tutorialspoint.com/cplusplus/cpp_switch_statement.htm • Slide 10 - Path coverage example: http://www.cs.odu.edu/~cs252/Book/branchcov.html • Slide 12 – Condition Coverage Example http://en.wikipedia.org/wiki/Code_coverage • Slide 15 – Apollo 13 mobie clip https://www.youtube.com/watch?v=C2YZnTL596Q • Slide 16 Getting Started http://blog.firespring.com/wp-content/uploads/2013/07/3-Steps-for- Getting-Started-with-Social-Media.jpg • Slide 17 - Documenting Processes http://pmtips.net/wp-content/uploads/2011/09/project_issues- 283x300.jpg • Slide 18 – Next Steps http://thebusinessupdate.co.za/wp-content/ uploads/2014/07/Steps-to-Starting-Forex-Trading-the- Right-Way.jpg