Log-Based Slicing for System-Level Test Cases
0 likes479 views
DS3 is a technique that decomposes complex system test cases into simpler test cases using log-based program slicing. It performs the following steps: 1. Uses static slicing to initially decompose test cases based on assertions. 2. Analyzes test case execution logs to identify dependencies between statements and global resources. 3. Refines the static slices by adding missing dependencies identified in the logs. 4. Merges slices that share all non-assertion statements to minimize duplication. An evaluation on two large systems found that DS3 produced well-formed test case slices and significantly reduced test case execution times compared to the original complex test cases and standard static slicing. The sliced test cases also maintained similar
![7
New Idea: Leaveraging Existing Logs to Refine Static Slices
Approach
• There are test case execution logs, obtained from past regression testing sessions
• The logs include run-time information about the system under test
• There is a traceability between test case statements and log messages (if not, test cases can be instrumented)
• The log contains information on the usage of global resources (if not, a watchdog process can be used)
• We can extract the global resources accessed (e.g., files, network connections) and the actions
performed (e.g., read/write files, open/close connections) upon executing each test statement
...
20210526:10:00:01 INFO [test_example.py:line 1] write ‘output.csv’
20210526:10:00:01 INFO [test_example.py:line 1] fdm initialized
20210526:10:00:02 INFO [test_example.py:line 2] read ‘output.csv’
20210526:10:00:05 INFO [test_example.py:line 3] read ‘output.csv’
...
Example log](https://image.slidesharecdn.com/2021-issta-ds3-final-210714181416/85/Log-Based-Slicing-for-System-Level-Test-Cases-7-320.jpg)
![10
Step 2: Def-Use Analysis for Global Resources
Approach
Based on the execution logs
...
20210526:10:00:01 INFO [test_example.py:line 1] write ‘output.csv’
20210526:10:00:01 INFO [test_example.py:line 1] fdm initialized
20210526:10:00:02 INFO [test_example.py:line 2] read ‘output.csv’
20210526:10:00:05 INFO [test_example.py:line 3] read ‘output.csv’
...
Example log
(line 1, ‘output.csv’, def)
(line 2, ‘output.csv’, use)
(line 3, ‘output.csv’, use)
Identify (statement, global resources, def/use)
…
Analysis Results
based on keywords
Lines 2 and 3 depends on (à) line 1](https://image.slidesharecdn.com/2021-issta-ds3-final-210714181416/85/Log-Based-Slicing-for-System-Level-Test-Cases-10-320.jpg)
Log-Based Slicing for System-Level Test Cases
- 1. Log-Based Slicing for System-Level Test Cases Salma Messaoudi1, Donghwan Shin1, Annibale Panichella1,2, Domenico Bianculli1, and Lionel Briand1,3 1 2 3
- 2. 2 Regression Testing Introdution • Arguably one of the most important activities in software testing • However, its cost-effectiveness can be largely impaired by the cost of individual test cases Quality assurance technique applied when changes are merged to an existing codebase Test Case x Test Case y Test Case z 2.5 hours 4 hours 2 hours Even if a perfect test case prioritization technique is applied, no faults could be detected during the first few hours
- 3. 3 Observation: Complex System Test Cases Introduction • Often very expensive (e.g., took 1h to run a single test case) • Why? • Trigger (and wait for) physical components that take a lot of time to complete • Poorly designed (e.g., containing multiple test scenarios combined into a single test case) From a collaborative industrial research project • The execution time of the decomposed test cases would decrease • The cost-effectiveness of test case prioritization could improve • Furthermore, providing finer-grained information about test results would facilitate debugging activities, such as fault localization What if we decompose complex system test cases?!
- 4. 4 Simple Idea: Program Slicing on System Test Cases Introduction • Decompose a complex system test case containing multiple test scenario into separate ones, each of them with only one test scenario and its corresponding assertions Static Slicing on line (4) Static Slicing on line (7) Decomposed Test Case 1? Decomposed Test Case 2? def test_example(): (1) fdm = create_fdm_setup() (2) ref = read_csv(‘output.csv’) (3) sim = deploy_proc(‘output.csv’) (4) self.assertEqual(ref, sim) (5) new = run_ic() (6) diff = FindDiffs(ref, new, 1E-8) (7) self.assertEqual(len(diff), 0) Simplified Example System Test Case
- 5. 5 Limitation of Static Slicing Introduction ‘output.csv’ is written by line (1) and read by lines (2) and (3) Static Slicing on line (4) Not executable due to the missing “hidden” dependency! def test_example(): (1) fdm = create_fdm_setup() (2) ref = read_csv(‘output.csv’) (3) sim = deploy_proc(‘output.csv’) (4) self.assertEqual(ref, sim) (5) new = run_ic() (6) diff = FindDiffs(ref, new, 1E-8) (7) self.assertEqual(len(diff), 0) Example System Test Case def test_static_01(): (2) ref = read_csv(‘output.csv’) (3) sim = deploy_proc(‘output.csv’) (4) self.assertEqual(ref, sim) Sliced Test Case
- 6. 6 Other Program Slicing Approaches? Introduction • Dynamic slicing* • Require to instrument the source code and to collect coverage information • Not feasible for a system composed of 3rd-party components for which the source code is not available • Does not address the problem of “hidden” dependencies as they are not captured by code coverage • Observation-based slicing** • Require to run each system test case multiple times • Not applicable for time-consuming system test cases (e.g., the ones developed by our industrial partner) * Bogdan Korel and Janusz Laski. 1988. Dynamic program slicing. Information processing letters 29, 3 (1988), 155–163 ** David Binkley, Nicolas Gold, Mark Harman, Syed Islam, Jens Krinke, and Shin Yoo. 2014. ORBS: Language-independent program slicing. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 109–120.
- 7. 7 New Idea: Leaveraging Existing Logs to Refine Static Slices Approach • There are test case execution logs, obtained from past regression testing sessions • The logs include run-time information about the system under test • There is a traceability between test case statements and log messages (if not, test cases can be instrumented) • The log contains information on the usage of global resources (if not, a watchdog process can be used) • We can extract the global resources accessed (e.g., files, network connections) and the actions performed (e.g., read/write files, open/close connections) upon executing each test statement ... 20210526:10:00:01 INFO [test_example.py:line 1] write ‘output.csv’ 20210526:10:00:01 INFO [test_example.py:line 1] fdm initialized 20210526:10:00:02 INFO [test_example.py:line 2] read ‘output.csv’ 20210526:10:00:05 INFO [test_example.py:line 3] read ‘output.csv’ ... Example log
- 8. 8 Our Approach: Decomposing System teSt caSes (DS3) Approach System Test Case System Test Cases Logs Candidate Sliced Test Cases Static Slicing Perform static slice on assertions 1 Def-Use Info for Global Resources Def-Use Analysis for Global Resources Identify (stmt, entity, def/use) 2 Refined Sliced Test Cases Log-based Slice Refinement Add missing statements 3 Slice Minimization 4 Final Sliced Test Cases DS3
- 9. 9 Step 1: Backward Static Slicing Approach Using standard static slicing approaches def test_example(): (1) fdm = create_fdm_setup() (2) ref = read_csv(‘output.csv’) (3) sim = deploy_proc(‘output.csv’) (4) self.assertEqual(ref, sim) (5) new = run_ic() (6) diff = FindDiffs(ref, new, 1E-8) (7) self.assertEqual(len(diff), 0) Example System Test Case def test_static_01(): (2) ref = read_csv(‘output.csv’) (3) sim = deploy_proc(‘output.csv’) (4) self.assertEqual(ref, sim) Sliced Test Case 1 (based on line 4) def test_static_02(): (2) ref = read_csv(‘output.csv’) (5) new = run_ic() (6) diff = FindDiffs(ref, new, 1E-8) (7) self.assertEqual(len(diff), 0) Sliced Test Case 2 (based on line 7)
- 10. 10 Step 2: Def-Use Analysis for Global Resources Approach Based on the execution logs ... 20210526:10:00:01 INFO [test_example.py:line 1] write ‘output.csv’ 20210526:10:00:01 INFO [test_example.py:line 1] fdm initialized 20210526:10:00:02 INFO [test_example.py:line 2] read ‘output.csv’ 20210526:10:00:05 INFO [test_example.py:line 3] read ‘output.csv’ ... Example log (line 1, ‘output.csv’, def) (line 2, ‘output.csv’, use) (line 3, ‘output.csv’, use) Identify (statement, global resources, def/use) … Analysis Results based on keywords Lines 2 and 3 depends on (à) line 1
- 11. 11 Step 3: Log-based Slice Refinement Approach To add missing “hidden” dependencies into static slices def test_static_01(): (2) ref = read_csv(‘output.csv’) (3) sim = deploy_proc(‘output.csv’) (4) self.assertEqual(ref, sim) Sliced Test Case 1 (based on line 4) def test_static_02(): (2) ref = read_csv(‘output.csv’) (5) new = run_ic() (6) diff = FindDiffs(ref, new, 1E-8) (7) self.assertEqual(len(diff), 0) Sliced Test Case 2 (based on line 7) def test_static_01(): (1) fdm = create_fdm_setup() (2) ref = read_csv(‘output.csv’) (3) sim = deploy_proc(‘output.csv’) (4) self.assertEqual(ref, sim) Refefined Sliced Test Case 1 def test_static_02(): (1) fdm = create_fdm_setup() (2) ref = read_csv(‘output.csv’) (5) new = run_ic() (6) diff = FindDiffs(ref, new, 1E-8) (7) self.assertEqual(len(diff), 0) Refined Sliced Test Case 2 Lines 2, 3 à line 1 Line 2 à line 1
- 12. 12 Step 4: Slice Minimization Approach Based on our observations in real-world codebases Observation: If all non-assertion statements of a slice are also in another slice, both slices share the same test scenario, and therefore should be merged def test_static_01(): (1) fdm = create_fdm_setup() (2) ref = read_csv(‘output.csv’) (3) sim = deploy_proc(‘output.csv’) (4) self.assertEqual(ref, sim) Refefined Sliced Test Case 1 def test_static_02(): (1) fdm = create_fdm_setup() (2) ref = read_csv(‘output.csv’) (5) new = run_ic() (6) diff = FindDiffs(ref, new, 1E-8) (7) self.assertEqual(len(diff), 0) Refined Sliced Test Case 2 Not a subset of the other The slices are final!
- 13. 13 Evaluation Evaluation • Aim to evaluate following aspects of DS3: • Slicing effectiveness (compared to vanilla static slicing) • Execution time of sliced test cases (compared to its original) • Test effectiveness of sliced test cases (compared to its original) • Evaluation subjects Programming Language Number of Test Cases Prop Python 30 JSBSim C++ and Python 81 (76) Table 1. Evaluation Subjects Summary
- 14. 14 Research Questions Evaluation • RQ1: How effective is DS3 in slicing system test cases compared to standard static slicing? • RQ2: How efficient are the slices produced by DS3 compared to the original test cases? • RQ3: What is the code coverage and fault detection capability of the slices produced by DS3 compared to the original test cases?
- 15. 15 RQ1: Slicing Effectiveness Evaluation • The standard static slicer has significantly lower slicing effectiveness than DS3 • Leveraging the global resource usages reported in the logs, DS3 is able to generate well-formed slices in all cases System # TCs Appr. # Slices SlicingEff (Pass/Total) Total Pass Fail Prop 30 DS3 137 137 0 1.00 Static Slicer 166 40 126 0.24 JSBSim 76 DS3 84 84 0 1.00 Static Slicer 169 56 113 0.33 Table 2. Slicing Effectiveness Comparision Results
- 16. 16 RQ2: Execution Time Evaluation • On average, the execution time of a sliced test case (239.6s) is much less than that of the original test case (3196.9s) • There are some cases where the total execution time of all slices for a system test case is greater than that of the original test case • Because there are the same “fixtures” shared by multiple slices • For some cases (e.g., TC30), even the total execution time of all sliced test cases is much less than that of the original • Because the original has “expensive” statements that actually do not have any dependencies on others System Test Case #Slices Execution Time (s) Original S-Avg S-Total TC1 2 1139 520.0 1040 TC2 2 498 342.5 685 TC3 3 245 132.6 398 TC4 1 330 201 201 … … … … … TC29 10 680 99.1 991 TC30 5 23231 200.8 1004 Average 4.9 3195.9 239.6 967.1 Table 3. Execution Time Results
- 17. 17 RQ3: Test Effectiveness Evaluation • There is no significant difference, meaning sliced test cases are as effective as their original test cases in terms of coverage and fault detection • The minor difference is because, again, some of the original test cases include statements that actually do not have any dependencies on others Test Case Covered Branches Covered Functions Killed Mutants JSBSim Original 5736 1831 289 Sliced 5698 1831 288 Table 4. Test Effectiveness Results
- 18. 18 Conclusion Conclusion • DS3 can automatically slice expensive system test cases into far less expensive sliced test cases • Slicing test cases rarely decrease the effectiveness compared to their original test cases • Sliced test cases can be further utilized by, for example, test case prioritization and test suite selection, to increase the cost-effectiveness of regression testing
- 19. 19 Log-Based Slicing for System-Level Test Cases @SnT_uni_lu SnT, Interdisciplinary Centre for Security, Reliability and Trust Connect with us Donghwan Shin Research Scientist [email protected] Salma Messaoudi, Donghwan Shin, Annibale Panichella, Domenico Bianculli, and Lionel Briand Contact: