On the Distribution of Test Smells in Open Source Android Applications: An Exploratory Study
0 likes151 views
This exploratory study investigates the prevalence and distribution of test smells in open-source Android applications, revealing that 97% of analyzed apps contain test smells, with assertion roulette being the most common. The research extends the catalog of known unit test smells and introduces an open-source detection tool, tsdetect, which identifies 19 types of test smells. Findings suggest that test smells are introduced early in the codebase and persist throughout the application's lifecycle.
On the Distribution of Test Smells in Open Source Android Applications: An Exploratory Study
- 1. On the Distribution of Test Smells in Open Source Android Applications: An Exploratory Study Anthony Peruma, Khalid Almalki, Christian D. Newman, Mohamed Wiem Mkaouer, Ali Ouni, Fabio Palomba 29th Annual International Conference on Computer Science and Software Engineering
- 3. Software maintenance is not cheap! 3 I n t r o d u c t i o n ▸A high-quality system need not be necessarily maintenance-friendly ▸Systems built using poor design/coding practices can meet functional requirements ▸In the long run, such events impact software maintenance - and maintenance is not cheap! ▹ Maintenance consumes 50% to 80% of resources
- 4. Towards maintenance-friendly code 4 I n t r o d u c t i o n ▸Researchers and industry have defined and created approaches and tools to detect code in need of refactoring ▹ Design/code smells - Cohesion, Coupling, God Class, etc. ▹ Tools - FindBugs, PMD, Checkstyle, etc. ▸Smells make code harder to understand and make it more prone to bugs and changes ▸Research and tools have been primarily on production code
- 5. Test Smells 5 I n t r o d u c t i o n ▸Test code, like production code, is subject to smells ▸Formally introduced in 2001 with 11 smell types ▸Inclusion of additional smell types through the years, analysis of their evolution and longevity, and elimination patterns ▸Tools to detect specific smell types ▸Studies on traditional Java applications
- 6. “2.6 million apps available on Google Play as of Q4 2018
- 7. Objective Insight into the unit testing practices of Android app developers with the aim of providing developers a mechanism to improve unit testing code 7 I n t r o d u c t i o n
- 8. Contribution 8 I n t r o d u c t i o n & O b j e c t i v e s Open-Source Test Smell Detection Tool Understanding of Test Smells in Android apps Expansion of Test Smell Types Replication Package Availability
- 9. Research Questions 9 I n t r o d u c t i o n RQ 02 What is the general trend of test smells in Android apps over time? ▹ When are test smells first introduced into the project? ▹ How do test smells exhibited by the apps evolve over time? RQ 01 How likely are Android apps to contain unit test smells? ▹ Are apps, that contain a test suite, prone to test smells? ▹ What is the frequency and distribution of test smells in apps? ▹ How does the distribution of smell types in Android apps compare against traditional Java applications?
- 10. 2. Test Smells
- 11. Proposed Test Smells 11 T e s t S m e l l s ▸ Conditional Test Logic ▸ Constructor Initialization ▸ Default Test ▸ Duplicate Assert ▸ Empty Test ▸ Exception Handling ▸ Ignored Test ▸ Magic Number Test ▸ Redundant Print ▸ Redundant Assertion ▸ Sleepy Test ▸ Unknown Test
- 12. Practicability 12 T e s t S m e l l s 120smelly unit test files 100software systems 120software developers 41.7%response rate Are our proposed smells indicative of problems?
- 13. Conditional Test Logic 13 T e s t S m e l l s ▸ Conditions within the test method will alter the behavior of the test and its expected output ▸ Developers agree on the negative impact on code comprehension ▸ However, outright removal may not always be applicable – decide on a “case by case basis” I actually have no idea why that for loop is there. It doesn’t do anything but run the test 1000 times, and there’s no point in that. I’ll remove it.
- 14. Constructor Initialization 14 T e s t S m e l l s ▸ Initialization of fields should be in the setUp() method (i.e., test fixtures) ▸ Most developers are aware of test fixtures ▸ Developers unanimously agree on using test fixtures ▸ Reasons for not using test fixtures include “laziness” and being “sloppy” I have already made this change since you pointed it out so the code is clearer now
- 15. Default Test 15 T e s t S m e l l s ▸ Default test class meant to serve as an example ▸ Should either be removed ▸ A test-first approach will force developers to remove the file ▸ Unanimous agreement among developers that the file “serves no concrete purpose” and that it may lead to confusion Removed useless example unit test
- 16. Duplicate Assert 16 T e s t S m e l l s ▸ The same condition is tested multiple times within the same test method ▸ The name of the test method should be an indication of the test ▸ Mixed responses - some developers preferred to split the assertion statement into separate methods I might enforce it on some bigger projects
- 17. Empty Test 17 T e s t S m e l l s ▸ When a test method has no executable statements ▸ JUnit will indicate that the test passes even if there are no executable statements present in the method body ▸ Unanimous agreement among developers that such test methods should be removed from the test suite Yes definitely should be removed
- 18. Exception Handling 18 T e s t S m e l l s ▸ Passing or failing of a test method is explicitly dependent on the production method throwing an exception ▸ Developers should utilize JUnit’s exception handling features to automatically pass/fail
- 19. Ignored Test 19 T e s t S m e l l s ▸ Ignored test methods result in overhead with regards to compilation time and an increase in code complexity and comprehension ▸ Mixed responses - investigate problems or serve as a means for new developers “to understand behavior” would not tolerate to have ignored tests in the code
- 20. Magic Number Test 20 T e s t S m e l l s ▸ Test method contains unexplained and undocumented numeric literals ▸ Developers agree that the use of constants over magic numbers improve code readability/understandability ▸ Not a blanket rule - a constant should only be used so that its “name adds useful information” If the numerical value has a deeper meaning (e.g. flag, physical constant, enum value) then a constant should be used.
- 21. Redundant Assertion 21 T e s t S m e l l s ▸ Assertion statements that are either always true or false ▸ Common reason for the existence of this smell is due to developer mistakes ▸ Developers confirmed that such code “is not needed”, “bad style” and “should probably be removed” ▸ Might exist to support edge cases
- 22. Redundant Print 22 T e s t S m e l l s ▸ Unit tests are executed as part of an automated script ▸ They can consume computing resources or increase execution time ▸ Unanimous agreement that print statements do not belong in test suites ▸ A common reason for the existence of this smell is due to developer debugging a waste of resources (cpu+disk space)
- 23. Sleepy Test 23 T e s t S m e l l s ▸ Explicitly causing a thread to sleep can lead to unexpected results as the processing time for a task differs when executed in various environments and configurations ▸ Developers confirmed that there are risks (i.e., inconsistent results) involved with causing a thread to sleep the alternative requires more code
- 24. Unknown Test 24 T e s t S m e l l s ▸ The assertion statement helps to indicate the purpose of the test ▸ JUnit will show the test method as passing ▸ Majority of the developers are in favor of having assertion statements in a test method ▸ Missing assertions were due to mistakes It looks like just sloppy coding there. I'll look to fix that test
- 25. TSDetect ▸ Open-source, Java-based, static analysis ▸ Available as a standalone jar and requires a list of file paths as input ▸ Utilizes an abstract syntax tree to parse and detect test smells ▸ Detects 19 test smells (12 proposed + 7 existing) ▸ Average F-Score of 96.5% 25 T e s t S m e l l s High-level architecture of TSDetect
- 27. Data Collection Phase 27 E x p e r i m e n t M e t h o d o l o g y F-Droid Repositories Data Mining Tools Mining Output 2,011cloned apps 1,037,236 commits 6,379,006 java files affected by commits +3.5 GB java files collected
- 28. Detection Phase 28 E x p e r i m e n t M e t h o d o l o g y Test File Detection Tool Test Smell Detection Tool Syntactically correct test files with 1 or more test methods Detected test smells 656analyzed apps 206,598 detected test files 1,187,055 analyzed test methods 175,866 test files with 1 or more smells
- 30. RQ1 – Test Smell Occurrence Test Smell Occurrence & Distribution ▹ 97% of the analyzed apps contained test smells ▹ Assertion Roulette occurred the most (in over 50% of the analyzed apps and test files) ▹ All smell types had a high co-occurrence with Assertion Roulette ▹ Similar distribution of test smells between Android and non-Android applications 30 A n a l y s i s & D i s c u s s i o n
- 31. RQ1 – Test Smell Occurrence 31 0.00% 2.00% 4.00% 6.00% 8.00% 10.00% 12.00% 14.00% 16.00% 18.00% S m e l l T y p e D i s t r i b u t i o n S m e l l T y p e O c c u r r e n c e A n a l y s i s & D i s c u s s i o n
- 32. RQ1 – Test Smell Occurrence 32 A n a l y s i s & D i s c u s s i o n S m e l l T y p e C o - O c c u r r e n c e
- 33. RQ2 – Test Smell Trend 33 A n a l y s i s & D i s c u s s i o n Test Smell Introduction ▹ The first inclusion of a smelly file occurs approximately 23% of the way through the total app commits ▹ A test file is added with 3 smell types ▹ Assertion Roulette is the frequently the first smell type introduced ▹ Smells exhibited by a file remains constant throughout all updates to the file
- 34. 5. Conclusion
- 35. Summary 35 C o n c l u s i o n ▸ Extended the catalog of known unit test smells ▸ Open source test smell detection tool ▸ A study of 656 Android apps showed a high prevalence of test smells in test suites ▸ Smells are introduced early on into the codebase and exist during the lifetime of the app ▸ Comprehensive project website: https://testsmells.github.io