Test automation workshop slideset
Download as PPT, PDF1 like412 views
The document discusses a workshop on test automation. It includes an agenda for the workshop covering topics such as what kind of test automation system participants would like to have, where testing time is spent, how to choose a test automation tool, and what is the right level of test automation for an organization. The workshop uses a format of presenting questions, having group discussions, and then presenting prepared material to promote learning.
Test automation workshop slideset
- 1. Test Automation Pondering Workshop for FYI on Winter 2011 [email protected] Towo Toivola Attribution (Tekijä mainittava) http://creativecommons.org/licenses/by/3.0/
- 2. Welcome to the Workshop I hope we all have an appetite for arguing and learning about test automation The slides are in English just to be sure, but I will present in Finnish if the audience prefers it.
- 3. Many Thanks Much of this material is heavily indebted to Maaret Pyhäjärvi, Erkki Pöyhönen, Petri Kuikka, Risto Kumpulainen, Dean Leffingwell.. Thanks also go to my friends at work and in the software engineering community Towo Toivola 2009
- 4. About Me Experiences define viewpoints MSc (eng) from HUT In F-Secure Corp for 11 years ◦ COTS software, custom systems ◦ Many many projects, large and small A year in Kosovo ◦ IT administration, system building Seminars, workshops Interactive speaker Towo Toivola 2009
- 5. About You How many are test engineers? How many managers? Any programmers? How many have some experience on testing tools? What is your industry? What are your most important requirements for this workshop? Towo Toivola 2009
- 6. Disclaimer As with everything, nothing is absolute I shall not discuss load-testing much, or test process automation Not an expert of the particular tools Your terminology may differ Attending this course may not be enough It may be that you should not do test automation There are two worlds, I will try to balance in between them Towo Toivola 2009
- 7. Agenda for Today 0900 Welcome (this) 1255 How would you 0920 What kind of choose your TA tool? TA system would you 1345 What is the right like? level for doing TA in 1000 What is the your organization? goal of your TA 1430 How to take TA Break initiative? into use? Break 1105 Where does all 1530 How will you the testing time go? change your products to ease TA? 1600 Resourcing TA in Lunch the long run? Towo Toivola 2009
- 8. About the Schedule There is a lot of ground to cover This is a new format, bear with me We can change things You can check what I am planning to leave out.. There should be enough breaks, we should be able to concentrate well on work ◦ What rules should we have? Towo Toivola 2009
- 9. Format for this Workshop We will use this format to deal with the questions in the agenda: Presenting a question (5min) Pondering in small groups (15min) Discussion with all together (10min) Presenting of prepared material (10min) This should make it into about 30-50 mins per problem if we don't get carried away.
- 10. Why this Format? Traditional teaching (providing answers) does not promote thought Thought and discovery promotes learning Struggling with a problem first and receiving guidance after that promotes understanding and application (I hope) Interaction promotes understanding and information dissipation I am trying to give you the most valuable things well rather than a lot of things Technical details are not as important as people think
- 11. Question #1 What kind of TA system would you like to have? (Outline your vision or what you want to have in one years time) 0920
- 12. Please take 15 to think about this First quickly by yourself Then within your group Document ideas, keywords and major agreements on provided office supplies Be prepared to present your views and findings, even if you are not so sure
- 13. Results and Discussion for Question #1 What kind of TA system would you like to have? (Outline your vision or what you want to have in one years time)
- 14. Why do You Want What You Want? Tools and technical solutions often dominate unduly in the decisions Organizational boundaries are often taken as granted and TA may lead to local optimization You have probably made a lot of implicit assumptions before even starting to actively think about the problem People tend to focus on fixing what they understand best instead of that which is most important.
- 15. Now that you have already discovered what should be the end result of today, we can start working in light of that assumption ;)
- 16. Question #2 What is the goal of your TA initiative? (Why are you here, why do you want to do TA, what do you expect to gain?) 1000
- 17. Group Work Time Firstquickly by yourself Then within your group Document Be prepared to present
- 18. Results and Discussion for Question #2 What is the goal of your TA initiative?
- 19. What’s the Use of Test Automation Cost savings ◦ Ability to run the same tests cheaper Faster development ◦ Ability to implement more features in the same amount of time Faster delivery ◦ Ability to do snap changes Better quality ◦ Ability to reach and maintain higher quality Towo Toivola 2009
- 20. Differences to Manual Testing Some tests that are manually possible are not possible to automate Some tests that are possible to automate are not possible to run manually Development takes much more time, execution much less Much more attention to detail, much less common sense Requires more maintenance Towo Toivola 2009
- 21. Enabling the Fast Feedback Loop A common and useful use of test automation Minimize the time from programming to quality information ◦ Code – test – fix – retest in hours VISIBILITY! ◦ If not in minutes! Continuous Integration Fully automatic tests and reporting ”The most important limiting factor for the velocity of an agile team” -DL Towo Toivola 2009
- 22. Regression and Automation VISIBILITY! Is the software still OK? Manual regression will get rid of your good test engineers Automation does not get bored The courage to do changes The ability to do quick releases Towo Toivola 2009
- 23. The Machine is Superior to Man Automation is faster Automation has more attention to detail Automation can run through more data Automation can execute more combinations Automation can measure more exactly Automation does not get tired Automation works through the night Automation provides exact logs Towo Toivola 2009
- 24. Man is Superior to the Machine Works around problems Common sense Creativity Understanding of the user Understanding of priorities Understands change Looking outside the box The most important bugs are found in the requirements and assumptions Towo Toivola 2009
- 25. What Do You Want? You Need to Prioritize, seriously If you don’t set clear goals, don’t expect clear benefits, or much results of any kind Test faster? Test more often? Test better? Test in new ways? Test cheaper? Towo Toivola 2009
- 26. Common Mistakes (that cost a lot of money) Saving a project Aiming for X% automated Working with untestable software Running a lot of tests, producing little value VISIBILITY! Organizational issues Skills of people working with automation #1: Unrealistic assumptions ◦ Automation will find loads of new bugs ◦ No real resourcing needed Towo Toivola 2009
- 27. Real Benefit of TA TA is only useful It does this.. when it provides More if you start information that.. early in the project Gets a bug fixed More if you run it Helps you make a constantly decision Only if you react to the results These Only if you believe require the results (re)action!
- 28. It's hard, but it's worth it. Test automation, when done sensibly, is hugely beneficial. It can give you decisive competitive advantage over your competitors both financially and in terms of employee motivation and satisfaction.
- 29. Break Time 1050 Towo Toivola 2009
- 30. Question #3 Where does all the testing time go? (Testing software is an expensive business, why does it cost so much? What do all those people spend all their time on?) 1105
- 31. Group Work Time Firstquickly by yourself Then within your group Document Be prepared to present
- 32. Results and Discussion for Question #3 Where does all the testing time go?
- 33. The Nature of a Test Which part can the automation handle? Towo Toivola 2009
- 34. Where Does Your Testing Time Go? Do you use too much time testing? Not enough time? Test planning? Test environment building? Regression testing? Testing new features? Testing large data collections? Testing with numerous environments? Towo Toivola 2009
- 35. Things You Should Automate Things that are run very often ◦ Core regression tests Things that are easy to automate and bring value Things that are tough to test manually ◦ Performance ◦ Race conditions ◦ Linear combinations ◦ Large amounts of data ◦ Reliability Things that have to be run quickly ROI !!!!11!!!! Towo Toivola 2009
- 36. Things You Should Not Automate Tests that are not important Tests that are easy to do manually and hard to automate Tests that are run seldom ◦ Unless they are critical or very hard to run manually Does-it-feel-right Tests where automation results are not reliable Towo Toivola 2009
- 37. But that is all the Sensible Stuff It is good to think about how to balance between explorative and regression testing, but I bet your testers also use time on.. Meetings, overhead, bureacracy Waiting (for fix, hardware, decision, info..) Arguing, pleading, fuming.. Testing wrong things Testing wrong builds..
- 38. In the Insensible Reality.. How much will making execution faster actually help? Should you look at the whole process? Maybe you should fix something else (as well)? Could you use TA implementation as a driver for new development culture?
- 39. Lunchtime! I have reserved one hour for lunch. 1155 Towo Toivola 2009
- 40. Question #4 How would you choose your TA tool? (Since you want to do TA, you need to somehow decide which tooling to use. How do you decide? How do you make the decision an informed one?) 1255
- 41. Group Work Time Firstquickly by yourself Then within your group Document Be prepared to present
- 42. Results and Discussion for Question #4 How would you choose your TA tool?
- 43. Finding a Tool Remember that a tool is not the same as test automation “A fool with a tool is a more dangerous fool” There are many vendors There are many free alternatives Tool finding must be considered real work ◦ With goals ◦ With resources Towo Toivola 2009
- 44. Cost, What is it? Many costs ◦ Cost of the tool (in the receipt) ◦ Cost of taking the tool into use (mainly hidden) Including maintenance ◦ Cost of not doing something else with the time ◦ Cost of maintenance You should consider all costs Cost of tool is usually overrated Towo Toivola 2009
- 45. Impact of Tool on Work Now, what is expensive? Big tool project Small tool project Cheap unsuitable tool Time Project end Towo Toivola 2009
- 46. My Advice Always take the one that is most suitable for you, taking into account: Cost of tool Intended use Personnel skills and attitudes Goals of the tool project Future plans for test automation Don't standardize for the sake of it Towo Toivola 2009
- 47. Plan the Tool Finding Make it a proper project If you still do projects Plan – set goals Resource explicitly Have many people join in Scope out the tools out there Select a couple for trials ◦ Do real work with them ◦ Not important, busy projects! Towo Toivola 2009
- 48. Remember You will need support in using the tool You will need changes to the tool Your software will change The platform your software will run on will change as well You may need changes to your software Or your development process.. Towo Toivola 2009
- 49. What Should it be Like? Relatively easy to take into use Enables automatic testing, not just playing with automation ◦ Including reporting Supports your technology well Does not seriously restrict your testers Can be run by a single person and as a central system Fits your organization Towo Toivola 2009
- 50. Or do You Need a Tool? You could make one, afterall you make software Just right for you Intergrates nicely with your product and development process You could just use common utilities to create a mash-up tool Build server SSH, remote exec Common scripting languages (with TA libraries..)
- 51. In a Modern, Efficient, Agile Organization In a galaxy far, far away.. The best people to make the trade-off decisions between tools are seasoned software development engineers who work in agile teams that have holistic responsibility for developing the software Trust your professionals to make the right choice and support them Their knowledge, interest and motivation are priceless in the implementation work of the TA Success will truly inspire others
- 52. Question #5 What is the right level for doing TA in your organization? (Should the SWE:s do UT/TDD? Should there be paired test engineers perfoming them? Or module tests? Should there be multi-tier integration tests? Can you do acceptance-level TA? Should you?) 1345
- 53. Group Work Time Firstquickly by yourself Then within your group Document Be prepared to present
- 54. Results and Discussion for Question #5 What is the right level for doing TA in your organization?
- 55. What is Module- and Unit testing? Module testing GUI Testing Tool environment Unit Testing Framework Graphical User Application Interface (GUI) Programming Interface (API) When significant When functionality is part of When each unit available from below can be used to functionality GUI or from side available in user catch the problems (components / modules interface. and their interfaces) “tester territory” “developer territory” Towo Toivola 2009
- 56. Can you Testing Levels find the silver bullet? Cost / System found bug Integration Module Unit Bug finding potential 0% 100% Towo Toivola 2009
- 57. Why Test Automation on Many Levels Faster automation Faster feedback loop ◦ Ability to do snap changes Better quality ◦ Ability to reach and maintain higher quality Faster (= cheaper) implementation Less maintenance work ◦ Again, = cheaper End result: Cost efficiency Towo Toivola 2009
- 58. Differences to GUI Automation API:s are more straightforward to use ◦ Implementation is faster and cheaper ◦ Tools are often free of charge API:s change much more seldom than a GUI ◦ Requires less maintenance Some tests that are possible through API:s are not possible to run using the GUI Some tests that are possible through the GUI are not possible to run using the API ◦ You will test a bit less of your application ◦ You can, probably, also module-test your GUI Running takes typically less time Code – test – fix – retest in minutes Detected bugs are usually easier to fix Towo Toivola 2009
- 59. Question #6 How to take TA into use? (Who should do it? Where do we get the resources? Do we have the skills? Which project? Which product? How to create the culture? How to ensure the long-term commitment of the organization?) 1430
- 60. Group Work Time Firstquickly by yourself Then within your group Document Be prepared to present
- 61. Results and Discussion for Question #6 How to take TA into use?
- 62. Slide of Truth About Everything Seriously, did you expect that I would have answers to that question for you? Some thoughts will follow, but remember that every case is unique.
- 63. Test Automation IS Software System Development Good test automation requires that you understand testing well, And software development And system implementation/administration And preferably test automation too. Most people don't reeeally get this Towo Toivola 2009
- 64. What Does That Mean? Source control and versioning Bug tracking Requirements definition Project management ◦ Iterative and incremental if you want to be successful Resourcing Testing! Fast feedback loop! Long-term commitment ETC. Towo Toivola 2009
- 65. Don’t Leave the 20% Undone ◦ Usability ◦ Reliability ◦ User base ◦ Significance ◦ Reporting ◦ Analysing problems ◦ Maintenance Towo Toivola 2009
- 66. Make sure you generate value ( = $$$$ ) not just run tests My serious main recommendation: One test case. Full chain. Beginning of project. VISIBILITY! Remember: TA is only useful when it provides information that.. Ugly slide.. Gets a bug fixed Helps you make a decision
- 67. Question #7 How will you change your products to ease TA? (Should you? How? Should you change your process? Why?) 1530
- 68. Group Work Time Firstquickly by yourself Then within your group Document Be prepared to present
- 69. Results and Discussion for Question #7 How will you change your products to ease TA?
- 70. A Word of Warning Automating an untestable software can be ruinously expensive Prepare for organization struggles when you need the software changed And the process Prepare for the organization to ignore any bugs found by automation unless they are reproduced by hand Towo Toivola 2009
- 71. Furthermore Developing untestable software is ruinous anyway, stop doing it There are N people changing the software, how will n TA people keep up? Who investigates the problems found in TA? Who fixes? The bugs in software? The changes needed for TA?
- 72. Question #8 Resourcing TA in the long run? (Who will make sure TA keep generating value next year and the one after? How? Changing software! Maintenance hell!) 1600
- 73. Group Work Time Firstquickly by yourself Then within your group Document Be prepared to present
- 74. Results and Discussion for Question #8 Resourcing TA in the long run?
- 75. Maintenance If you get your tests running fine, don’t think that’s most of the work done The rest of your R&D works constantly – to change your software Maintenance is in top 2 killers of test automation ◦ The other is organizational issues You must be able to maintain your test code, constantly and easily Towo Toivola 2009
- 76. Which Bit Would You Like to Maintain? Towo Toivola 2009
- 77. How the Math Goes.. You invest 3 people for automation You invest 2 months for finding the tool You invest 6 months for developing a good suite of tests Changes in the application make the tests unreliable, so they will not be run, the organization will not believe in bugs found by the system You just lost more than 3x(2+6)x5000e = 120 000e worth of investment I did not include the price of the tool.. Towo Toivola 2009
- 78. What Should We Learn Implementing a test automation system is implementing a constantly supported service system based on software Good system building and programming practises must be followed ◦ Duplicate code is your arch enemy Resources must be allocated to maintain the system at all times Towo Toivola 2009
- 79. Something I Have Learned Since the 90’s Model-based test automation technique Importance of visual controls Social mass is more important than technical mass Just do it 08/03/10 Towo Toivola 2009
- 80. Things we (probably) didn't have time for.. Technicaldetails Data-driven vs keyword-driven vs model- based Discussing individual tools Discussing how to make software in general Much else..
- 81. Debrief of the Day What do we remember from today? Was this day useful? ◦ What needs to be improved? ◦ What was best? What will you do differently in the future? BRoA #53 If you learn something and change nothing, you didn't learn anything. The topic is free, let’s have a discussion! Towo Toivola 2009
- 82. Thank You for Your Time We made it to the finish! Towo Toivola 2009
- 83. Next is a bunch of reserve slides that may come in handy at some point...
- 84. Robustness Automation scripts must run reliably When not testing something you should perform operations in the most robust way possible Error handling logic Duplicate code causes unreliability and maintenance hell Common operations belong to libraries ◦ Maintained, high-quality Towo Toivola 2009
- 85. Libraries Required quality Utility libraries (handful) Use-case libraries Amount (handful) maintenanc Test drivers e Test case (handful) scripts (dozens or hundreds) Testing value Towo Toivola 2009
- 86. Robustness example Towo Toivola 2009
- 87. Data-Driven If automation can do something, it makes sense to do it a lot Loop over data, run combinations ◦ Example: Log in with 10 incorrect accounts and many correct accounts Towo Toivola 2009
- 88. Keyword-Driven Basically a new language, just for your testing Reading in commands and data instructions from a simple, human readable format Enables non-technical people to understand and generate test cases ◦ Bring in the business knowledge Requires quite a bit of work Towo Toivola 2009
- 89. (a very simple) Example Go: EM_mainscreen Do: Enter message: “Hello work” Go: Logout Go: EM_mainscreen Do: Verify message: “Hello work” Towo Toivola 2009
- 90. ATDD Acceptance/automated test driven development Agree on the test first, on a level understandable for all Keywords implement the test Software is developed to meet the test Highly advanced development / automation method Towo Toivola 2009
- 91. (a very simple) Example Given at effort manager login screen When login with incorrect account Then incorrect login reported Towo Toivola 2009
- 92. Model-Based Testing Creating a state-machine that models (some parts) of your software Verifying the software behaves according to the state-machine Combining transitions and data in many ways ◦ Different algorithms exist ◦ Random is pretty effective Challenging, very advanced Towo Toivola 2009
- 93. (a very simple) Example State: emanager-login-screen Input: login-incorrect Resultstate: badlogin-complaint Input: login-correct Resultstate: emanager-mainscreen State: emanager-mainscreen Input: logout Resultstate: emanager-login- screen … Towo Toivola 2009
- 94. Automatic Testing Big difference between computer aided testing and automatic testing Automatic is automatic, human intervention is not needed ◦ Starts automatically ◦ Runs automatically ◦ Analyses results automatically ◦ Reports automatically Could your tests do that? Towo Toivola 2009
- 95. Exercise: Automatic Make your tests run so that they can be executed automatically • Cmdline usage of TestPartner • Driver Script We should have about a half an hour. Towo Toivola 2009
- 96. Reporting How would you like to get the test results? Opening the tool? Going to the DB? By email? On a web page? In a file? As SMS? In public view? Towo Toivola 2009