Towards the Third Wave of Software Testing

1. Towards the Third Wave of Software Testing Bestoun S. Ahmed (Ph.D. Software Engineering) Department of Mathematics and Computer Science Karlstad University, Sweden Phone: +46 54 700 1861 Room: 21D 413 [email protected] Docent Lecture Karlstad 8 April 2021

2. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Software Testing • Formal definition (IEEE): Testing, is an activity performed for evaluating product quality, and for improving it, by identifying defects and problems. • Software failures can lead to disastrous consequences 2 •Loss of data ….. • Loss of fortune ….. • Loss of lives …..

3. The first wave of software testing Test Automation

4. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. • Bringing automation to let the computer test your software. 4 From Manual to Automated Testing

5. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. • Bringing automation to let the computer test your software. 4 From Manual to Automated Testing Test Results System under Test Test Execution Requirements Test Plan Test Design Test Cases

6. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. • Bringing automation to let the computer test your software. 4 From Manual to Automated Testing Test Results System under Test Test Execution Requirements Test Plan Test Design Test Cases

7. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. • Bringing automation to let the computer test your software. 4 From Manual to Automated Testing Test Results System under Test Test Execution Requirements Test Plan Test Design Test Cases Requirements Test Results Test Execution System under Test Capture/ReplayTool Test Plan Test Cases Test Design Test Scripts

10. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. 5 Our Approach to Automation

11. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. 5 Soft Testing Our Approach to Automation

12. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. 5 Soft Testing Our Approach to Automation

13. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. 5 Model-based Testing Soft Testing Our Approach to Automation

17. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. 5 Model-based Testing Mathematical Objects Soft Testing Our Approach to Automation

21. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. 5 Model-based Testing AI methods Mathematical Objects Soft Testing Our Approach to Automation

27. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Modelling and Formalisation 6

28. “We Cannot test everything...” The Problem “Things might seem uncontrollable” The need for “Clever Sampling”

29. Moving forward to Model-basedTesting and AI The second wave of testing

30. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Automated Test Generation • We want to try all possible inputs, and see what happened, but .. • Exhaustive testing is impossible… So, we want to be smart and try the important inputs. 9 Input Software SoftwareTesting in Principle Find faults Measure the coverage of path, lines, structure … etc. Cover combinations of inputs Exercise the user behaviour Measure any quality characteristics Goal of Testing Measure

31. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. The Search Problem • With the smallest set of inputs (test suite), I want to: - Cover all combinations of t-inputs (t-wise) - Cover most of the possibilities of input categories - Cover most of the code branches - Confirm that a property is working as the customer want. - … etc. 10 Measure How many faults we found? How many branches, or Lines of Code we covered? How many combinations we covered? Goal of Testing … Search & Optimise Test suite

32. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. The Measure (Objective Function) • If we can measure a goal of testing, we can assume it as an objective function to be satisfied while we generate a test suite. • This is the base of automated test generation. • We can use heuristics to search for best solution automatically. 11 Goal of Testing Satisfied? Search Best Achieved Test Suite No Yes • If we can measure a goal of testing, we can assume it as an objective function to be satisfied while we generate a test suite.

33. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12

34. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 Iteration I

35. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 Iteration I

36. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Iteration I

37. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set Iteration I

38. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set Iteration I

39. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set Iteration I Iteration I+1

40. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set In1 In2 … Int Iteration I Iteration I+1

41. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set In1 In2 … Int Iteration I Manipulate according to best achievement so far Iteration I+1

42. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set In1 In2 … Int Iteration I Manipulate according to best achievement so far Iteration I+1

43. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set In1 In2 … Int Iteration I Manipulate according to best achievement so far Iteration I+1 Final Iteration

44. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set In1 In2 … Int Iteration I Manipulate according to best achievement so far Iteration I+1 Final Iteration

45. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set In1 In2 … Int Iteration I Manipulate according to best achievement so far In1 In2 … Int Iteration I+1 Final Iteration

46. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. How it works • Generate a random test suite with all possible test cases. • Score each test case based on the fitness function (i.e., the measure). • Update the test suite based on the best achieved test suite so far. • Iterate until no best test suite can be found. • The update mechanism comes from the metaheuristc will be used. 12 In1 In2 … Int Generate random test set In1 In2 … Int Iteration I Manipulate according to best achievement so far In1 In2 … Int Iteration I+1 Final Iteration Best achievement test suite

49. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. New Optimisation and Test Selection Methods 13

60. Developing new testing methodologies for software applications

61. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. New Testing Methodologies (1) • Using code structure to generate the test cases. • Mutation testing to generate test cases. • Process testing to consider path-based test selection • Business model testing. • Avocado: fully automated test framework 15

62. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. New Testing Methodologies (2) 16

70. UI Testing Moving forward with the MBT

71. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Automation in GUI Testing • Automation via model-based testing. • Model the even interaction between the user and the GUI application. • In the graph model we try to catch all the executed sequence of the GUI under test by modelling all possible events using the event flow graph (EFG). • The events that open or close menus, or open windows are considered which are not actually a testing event rather than they are event to start the actual events for testing. 18 • To overcome this issue, a refinement GUI modelling has been recently developed called event interaction graph (EIG) GUI EFG EIG Ref. Bestoun S. Ahmed, Mouayad A. Sahib, and Moayad Y. Potrus, ” Generating Combinatorial Test Cases Using Simplified Swarm Optimization (SSO) Algorithm for Automated GUI Functional Testing“, Engineering Science and Technology, an International Journal, 17 (4), Pages 218-226 , (2014) Elsevier

72. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Automation Mobile App Testing • Using revers engineering approach to generate UI model for the mobile apps. • Most of these tools are typically based on dynamic approaches where an application is dynamically analysed at the run-time to extract information. • Tool example, Android GUITAR, Android GUI Ripper, MCrawlT, and test automation system. • Most of these tools are pure black-box techniques that perform dynamic analysis of applications. 19 Ref. Ibrahim-Anka Salihu, Rosziati Ibrahim, Bestoun S. Ahmed, Kamal Z. Zamli, and Asmau Usman, ” AMOGA: A Static-Dynamic Model Generation Strategy for Mobile Apps Testing“, IEEE Access, 2019, Vol. 7, Issue 1, PP. 17158-17173

73. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Automation in Web app Testing • Using exploratory testing as a replacement of crawlers to construct a model fore the web app. • An exploration of the parts of the SUT that have not been previously explored and tested. • Tapir framework: uses model reconstruction of the system under test. 20 • Tapir Browser Extension: n tracks a tester’ s activity in the SUT and sends the required information to the Tapir headquarters (HQ) component. • Tapir HQ: implemented as a standalone web application that guides the tester through the SUT, provides navigational test cases. • Tapir Analytics: This component enables the visualisation of the current state of the SUT model. Ref. Miroslav Bures, Karel Frajtak, Bestoun S. Ahmed, “Tapir: Automation Support of Exploratory Testing Using Model Reconstruction of the System Under Test“, IEEE Transactions on Reliability, Volume: 67, Issue:2, Pages: 557-580, (2018)

74. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Mega-model construction—Web app testing 21

75. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Automation in Smart TV app Testing • There is an urgent need to have a testing framework for smart TV apps. • The framework constructs a model for a given smart TV app cumulatively by exercising the user interface and extracting the runtime information using the remote control device interaction with the app. • Black-box reverse engineering stage used to generate a comprehensive model for the app-under-test. • Using the FSM and directed graphs as a model to represent the widgets and events on the GUI leads to a graph with many nodes and edges. 22 Ref. Bestoun S. Ahmed and Miroslav Bures, “EvoCreeper: Automated Black-Box Model Generation for Smart TV Applications” Accepted for publication in IEEE Transactions on Consumer Electronics, 2019

76. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Automating Smart TV app Testing • Navigation problem using remote control device. 23

77. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Automated Smart TV app Testing - Mega and Sub models • Mega model is the first model constructed after automated crawler using the remote control device. • Sub-models generated automatically based on user customisation to separate the sub-model from the mega model. • This will support auto model-based prioritisation. 24 Mega-model Sub-models

78. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. New approaches for UI Testing 25

84. Moving forward from Application to System- levelTesting Towards the third wave of testing

85. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. IoT System Testing • Testing in Scale. • The need for new metrics to be considered for testing. • What are the quality attributes to be considered? • The relation between the software application and the system level attributes. 27

86. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. New IoT Testing Methodologies 28

93. Moving towards AI Engineering

94. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Data Science and Software Engineering 30

95. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Data Science and Software Engineering 30

96. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Data Science and Software Engineering 30 Software development lifecycle

97. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Data Science and Software Engineering 30 Software development lifecycle

98. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Data Science and Software Engineering • Data-centric challenges: • Incomplete Data Gathering • Inconsistent Data Storage • Misleading Decision Making • Identifying anomality in data 30 Software development lifecycle

99. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Data Science and Software Engineering • Data-centric challenges: • Incomplete Data Gathering • Inconsistent Data Storage • Misleading Decision Making • Identifying anomality in data 30 Software development lifecycle • ML-centric challenges: • ML model quality • Correctness verification. • Model degradation • Test generation for ML

100. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. AI Engineering • How to engineering the development process? • Seeing AI systems as Software Systems in production. • Following the same engineering approach for development. 31 Fig. 4. Research agenda for AI engineering *Graph Source: J. Bosch, I. Crnkovic, H. Holmström Olsson, Engineering AI Systems:A Research Agenda,Artificial Intelligence Paradigms for Smart Cyber-Physical Systems

101. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. AI Engineering and Software Testing 32 *Graph Source: Machine Learning in Production, https://ckaestne.github.io/seai/

105. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. AI Engineering and Software Testing 32 *Graph Source: Machine Learning in Production, https://ckaestne.github.io/seai/ ClassicalV&V for software Systems

108. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Our Vision • Data verification and training data evolvability • ML decision making correctness and algorithm testing • Testing for ML model degradation. • Training Data Evolvability - test the training data against the used model • Quality of the data: Insuﬃcient data, irrelevant features, non-representative training data, overfitting, under-fitting, outliers. 33

109. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. Evolution of Test Automation • The testing tools and technologies are evolving very fast. • The systems to be tested will change, there is a new direction to develop new techniques for testing - Cloud services, edge computing … - Data driven systems - Machine learning systems • Cloud testing more to be used as a cheap resource and easily paralleled. • Security testing become more critical. • More use of dashboards for test diagnosis and test reuse. • Testing in internet scale, building systems of systems and distributed systems. 34 *Graph Source: https://www.testim.io/blog/ai-transforming-software-testing/

110. Computer Science KARLSTAD UNIVERSITY Department of Computer Science Bestoun S. Ahmed, Ph.D. 36 Computer Science

Towards the Third Wave of Software Testing

More Related Content

Similar to Towards the Third Wave of Software Testing (20)

Recently uploaded (20)

Towards the Third Wave of Software Testing