Unit testing [4] - Software Testing Techniques (CIS640)
1 like166 views
The document presents an in-depth overview of unit testing, covering essential terms, concepts, and structures involved in writing tests, including test cases, test runners, and testing frameworks across different programming languages. It discusses the use of assertions, setups, and teardowns, as well as the implementation of parameterized unit tests and test doubles. Additionally, it highlights common patterns and best practices for effective unit testing to ensure software reliability.
![Parameterized Unit Tests
void TestAdd() {
ArrayList a = new ArrayList(0);
object o = new object();
a.Add(o);
Assert.IsTrue(a[0] == o);
}
- Parameterized Unit Tests by Nikolai Tillmann and Wolfram Schultz [FSE’05] (Section 2)](https://image.slidesharecdn.com/unit-testing-170926175035/85/Unit-testing-4-Software-Testing-Techniques-CIS640-22-320.jpg)
![Parameterized Unit Tests
void TestAdd() {
ArrayList a = new ArrayList(0);
object o = new object();
a.Add(o);
Assert.IsTrue(a[0] == o);
}
void TestAdd(ArrayList a, object o) {
if (a != null) {
int i = a.Count;
a.Add(o);
Assert.IsTrue(a[i] == o);
}
}
- Parameterized Unit Tests by Nikolai Tillmann and Wolfram Schultz [FSE’05] (Section 2)](https://image.slidesharecdn.com/unit-testing-170926175035/85/Unit-testing-4-Software-Testing-Techniques-CIS640-23-320.jpg)
![Parameterized Unit Tests
void TestAdd() {
ArrayList a = new ArrayList(0);
object o = new object();
a.Add(o);
Assert.IsTrue(a[0] == o);
}
void TestAdd(ArrayList a, object o) {
Assume.IsTrue(a != null);
int i = a.Count;
a.Add(o);
Assert.IsTrue(a[i] == o);
}
- Parameterized Unit Tests by Nikolai Tillmann and Wolfram Schultz [FSE’05] (Section 2)](https://image.slidesharecdn.com/unit-testing-170926175035/85/Unit-testing-4-Software-Testing-Techniques-CIS640-24-320.jpg)
Unit testing [4] - Software Testing Techniques (CIS640)
- 1. Unit Testing Venkatesh-Prasad Ranganath Kansas State University Slides with * in title capture whiteboard content
- 2. Unit Testing Terms • Test • UUT / SUT • Test case • Structure • Test Data and Test Logic • Assertions • Test Suite • Test Fixture • Setup • Teardown • Test Runner • Testing Framework
- 3. Unit Testing Terms • Test(ing) - The act of exercising software with test cases • UUT / SUT - Unit / System Under Test • Test Case - An identifiable unit of testing; often, bundled as a method/function • Structure - Setup, Execution, Verify, Teardown (Arrange, Act, Assert) • Test Data and Test Logic - Integral components of a test case • Assertion - Preferred mechanism to state/communicate test outcome if <actual outcome is not the same as expected outcome> then raise AssertionError() # test failed else pass
- 4. Unit Testing Terms • Test Suite - A collection of test cases • Test Fixture • Setup - Preparation to be performed before tests • Teardown - Cleanup to be performed after tests • Test Runner - A component that orchestrates the execution of test cases and collects the test verdicts. • Testing Framework - A framework that supports test automation; typically, includes test runner, reporting capability, coverage calculation, and other features.
- 5. XUnit Testing Frameworks in Different Languages Python • TestCar — test class name • test_drive — test method name • unittest, Nose, PyTest • Mostly based on convention and consistent practice Java • CarTest • testDrive • @Test — Test method annotation • JUnit, TestNG, Spock • Mostly based on convention and consistent practice Common wisdom in unit testing • Many test cases for one method/function • Many test classes for one class
- 6. Coding Session* checkId(x) Specification • x is a string • If not, raise TypeError • x represents a numerical value • If not, raise ValueError • True if x has 9 characters • False otherwise
- 7. Test Runner How does a test runner work?
- 8. Test Runner How does a test runner work? • find/discover test cases • based on rules or conventions or annotations • E.g., functions whose name starts with “test” • execute each test case • collect and compile the test verdicts
- 9. Testing Framework How do most XUnit testing framework determine test verdict?
- 10. Testing Framework How do most XUnit testing framework determine test verdict? • Failure if test throws exception X • Pass if test completes (without any exception / failures) • Error otherwise • Most XUnit framework rely on assert statements and use the corresponding exception (e.g., AssertionError) as exception X
- 11. Assertions • What's the deal with assertion? • What are the pros and cons of using assertions? • With regards to inputs/outputs, when to use assertion and when to use error handling?
- 12. Assertions What's the deal with assertion? • Assertion is a mechanism to check if (should-be- true) properties are true; typically, guarantees • E.g., input array to a binary search function should be sorted • E.g., output array returned by a sorting function should be of the same length as the input array What are the pros and cons of using assertions? • + Easily and succinctly add checks to programs • - Imposes performance penalties
- 13. Assertions With regards to inputs, when to use assertion and when to use error checking? (Guidance) • Use error checking to handle both valid and invalid inputs and generate corresponding outcome; when both valid and invalid inputs are expected. • E.g., sq_root will return the square root of the input integer if it is positive; otherwise, raise ValueError exception. • Input provider does not promise to ensure the validity of input. • Use assertions to check inputs are valid; when only valid inputs are expected. • E.g., sq_root will return square root of input integer and it should be invoked only with positive integers. • Input provider promises to provide the validity of input.
- 14. Test Case Structure Setup, Execution, Verify, Teardown (Arrange, Act, Assert) • What do we do in Execution? • What do we do in Verify? • What is the common code pattern to test for outcome and absence/presence of exceptional behavior? • Setup and Teardown are considered part of Test Fixture
- 15. Test Case Structure • What do we do in Execution? • Set up data/objects necessary to obtain actual outcome and observe the outcome • What do we do in Verify? • Compare the observed outcome to expected outcome
- 16. Test Case Structure What is the common code pattern to test for output? <action_1> <action_2> … output = <action_n> assert output == <expected_output> We use an appropriate operator in place of == to compare observed output and expected output
- 17. Test Case Structure What is the common code pattern to test for absence of exceptional behavior/outcome? try: <action_1> <action_2> … <action_n> except: assert False
- 18. Test Case Structure What is the common code pattern to test for presence of exceptional behavior/outcome? try: <action_1> <action_2> … assert False except: pass
- 19. Test Case Structure What is the common code pattern to test for presence of specific exceptional behavior/outcome, i.e., raises exception X? try: <action_1> <action_2> … assert False except X: pass except: assert False
- 20. Test Fixture What should we do in Setup? • ?? What should we do in Teardown? • ??
- 21. Test Fixture What should we do in Setup? • Put the UUT in the state required for testing • Perform actions common to a set of test cases What should we do in Teardown? • Perform clean up actions • Release external resources • Delete interim artifacts • Restore system back to a good state
- 22. Parameterized Unit Tests void TestAdd() { ArrayList a = new ArrayList(0); object o = new object(); a.Add(o); Assert.IsTrue(a[0] == o); } - Parameterized Unit Tests by Nikolai Tillmann and Wolfram Schultz [FSE’05] (Section 2)
- 23. Parameterized Unit Tests void TestAdd() { ArrayList a = new ArrayList(0); object o = new object(); a.Add(o); Assert.IsTrue(a[0] == o); } void TestAdd(ArrayList a, object o) { if (a != null) { int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o); } } - Parameterized Unit Tests by Nikolai Tillmann and Wolfram Schultz [FSE’05] (Section 2)
- 24. Parameterized Unit Tests void TestAdd() { ArrayList a = new ArrayList(0); object o = new object(); a.Add(o); Assert.IsTrue(a[0] == o); } void TestAdd(ArrayList a, object o) { Assume.IsTrue(a != null); int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o); } - Parameterized Unit Tests by Nikolai Tillmann and Wolfram Schultz [FSE’05] (Section 2)
- 25. Parameterized Unit Tests • PUTs are test methods/cases generalized by allowing parameters • Most often, parameters correspond to test data (as in example-based testing) • PUTs are not test cases • PUTs are templates of test cases • Test cases are instantiations of PUTs with specific arguments (inputs)
- 26. Test Structure Example-based Test Case 1. Setup 2. Execute 3. Verify 4. Teardown Parameterized Unit Tests 1. Check assumptions about parameters/inputs 2. Setup 3. Execute 4. Verify 5. Teardown
- 27. Test Doubles (Harness) What is a test double? - xUnit Test Patterns by Gerard Meszaros (Chapter 23)
- 28. Test Doubles (Harness) What is a test double? • SUT may depend on components (DOCs) that are not under test • Test doubles are replacements for DOCs - xUnit Test Patterns by Gerard Meszaros (Chapter 23)
- 29. Why use Test Doubles?
- 30. Why use Test Doubles? • DOC is overly large / long setup time • DOC is slow • Do not want to mess existing DOC • DOC is unavailable or not ready • Test dependency on DOC interface (not implementation)
- 32. Test Test Doubles (Harness) Setup Exercise Verify Teardown SUT DOC Test Double - xUnit Test Patterns by Gerard Meszaros (Chapter 23)
- 33. Test Doubles (Harness) What are the different sorts of test doubles? • Dummy • Fake • (Test) Stub • Test Spy • Mock Object - xUnit Test Patterns by Gerard Meszaros (Chapter 23)
- 34. Test Dummy Setup Exercise Verify Teardown SUT Dummy Create Install Query Empty methods that throw error if invoked - xUnit Test Patterns by Gerard Meszaros (Chapter 27) How do we specify the values to be used in tests when their only usage is as irrelevant arguments to SUT method calls? We pass an object that has no implementation as an argument of a method called on the SUT.
- 35. Test Fake Setup Exercise Verify Teardown SUT Fake Create and Initialize Install Query Data - xUnit Test Patterns by Gerard Meszaros (Chapter 23) How can we verify logic independently when depended-on objects cannot be used? How can we avoid slow tests? We replace a component that the SUT depends on with a much lighter-weight implementation. Data
- 36. Test Test Stub Setup Exercise Verify Teardown SUT Test Stub Create Install Query Return stock values Indirect input - xUnit Test Patterns by Gerard Meszaros (Chapter 23) How can we verify logic independently when it depends on indirect inputs from other software components? We replace a real object with a test-specific object that feeds the desired indirect inputs into the system under test.
- 37. Test Test Spy Setup Exercise Verify Teardown SUT Test Spy Create Install Query Capture output values for later checking Indirect output - xUnit Test Patterns by Gerard Meszaros (Chapter 23) How do we implement behavior verification? How can we verify logic independently when it has indirect outputs to other software components? We use a test double to capture the indirect output calls made to another component by the SUT for later verification by the test.
- 38. Test Mock Object Setup Exercise Verify Teardown SUT Mock Object Create Install Final verification Check output values and perform final check Indirect output - xUnit Test Patterns by Gerard Meszaros (Chapter 23) How do we implement behavior verification for indirect outputs of the SUT? How can we verify logic independently when it depends on indirect inputs from other software components? We replace an object on which the SUT depends on with a test-specific object that verifies it is being used correctly by the SUT.