How to make Automation an asset for Organization
- 1. How to Make Automation An Asset to the Organization Vipin Jain Metacube Software, Jaipur, India QA&TEST 2012 11th International Conference on Software QA and Testing October 17-19, 2012 • Bilbao Spain
- 2. Introduction • Software Testing is a process that consists of all test life cycle activities like static and dynamic testing concerned with planning, preparation and evaluation of software products to determine that the software products satisfy customers requirements and are fit for customers use. • Software Testing is done to find software defects or failures in advance. • Software testing can also be stated as the process of validating and verifying that a software program/application/product: – Meets the business and technical requirements that guided its design and development. – Works as expected
- 3. Manual Testing • Manual testing is the process of manually testing software for defects. It is a laborious activity that requires the tester to possess a certain set of qualities; to be patient, observant, speculative, creative, innovative, open-minded, resourceful, and skillful. Advantage of Manual Testing • Running the test case is less cost than automation. • It allows the tester to perform more Ad-hoc testing (1) (random testing). • More time for testing enables a tester to find more bugs. Disadvantage of Manual Testing • Running tests manually can be very time consuming • Each time there is a new build, the tester must re-run all required tests - which after a while would become very dull and tiresome.
- 4. Automation Testing • Software test automation refers to the activities and efforts that intend to automate engineering tasks and operations in a software test process using well-defined strategies and systematic solutions. • The major objectives of software test automation is to free engineers from tedious and redundant manual testing Operations. • To speed up a software testing process, and to reduce software testing cost and time during a software life cycle • To increase the quality and effectiveness of a software test process by achieving pre-defined adequate test criteria in a limited schedule • The major key to the success of software automation is to use a systematic solution to achieve a better testing coverage.
- 5. Why Automation? • Over the last decade, test automation has become a crucial part in the Test planning activities for various QA groups. • A well written automated test suite is of enormous help in daily testing activities, especially in today’s agile world. • With the Advent of Mobile technologies, a new dimension has been added in Software testing. There are millions of mobile applications flooding markets each day. They need to be tested effectively with shorter QA cycles. But, there are a billion combinations of hardware devices, OS’s, carriers, and networks today and traditional manual testing cannot cover all these scenarios – especially when the app to market life cycle has to be short. Automation is the need of the hour.
- 6. Advantages of Automation Reliable1 Repeatable2 Programmable3 Comprehensive4 Reusable5 Fast6 Better Quality Cost Reduction8 Software7
- 7. Why it fails and what factors contribute to its failure?
- 8. Reasons for Failure • Management unwillingness for it – Be it lack of vision, cost associated or any other issue, management doesn’t often give a nod for it. • Lack of Vision - There is no clear vision behind what the automation will do, what we intend to achieve with it and what are the strategies to do this. • Time - No time to develop and maintain the automation scripts • Cost associated - We need the tools for it and we want management to assign the appropriate budget, but they are not interested. • Skills needed – The tool is new and appropriate training and time to master the tool are required. • Lack of Automation Matrix - There are no clear factors listed against which we measure our automation results to label it as success or failure • Automation engineers – There is no in-betweens Developers and QA Engineers. Developers demoted to "test development", can backfire, especially if they do NOT have the mindset of what they're testing for.
- 9. What to Automate? • Use cases that are fully developed and with clear understanding should be automated first • Relatively stable areas of the application over volatile ones must be automated. • Automate the tests that are repetitive over multiple builds. • Tests that tend to cause human error. • Tests that require multiple data sets. • Frequently used functionality that introduces high risk conditions. • Manual tests that are virtually impossible to be carried out. • Tests that involves a variety of different hardware or software platforms and configurations. • Tests that take a lot of effort and time when testing manually
- 10. When should we do Automation? • To get maximum benefit, automation testing should be started as early as possible. It should be run as often as needed. It’s a well known fact that the earlier testers get involved in the life cycle of the project, the better. Moreover , the more you test, the more bugs you find. • Automated unit testing can be implemented on day one and then it should be grown into automated test suite. • Always remember that bugs detected early are a lot cheaper to fix than those discovered later in production or deployment. Hence Start early. TEST EARLY AND TEST OFTEN
- 11. A Software Test Automation Process Test Automation Planning Design Automation Evaluate tools and select the Strategies & Solutions best tool(s) Develop & Implement Test Automation Solutions Introduce and Deploy Test Automation Solutions Review and Evaluate Software Test Automation
- 12. Automation Myths/Realities Test Plans covering all resource No commercially available tool that requirements, time needed and can create a comprehensive test strategy can be auto-generated. plan. Any application can be tested using No single test tool exists that can the tool. be used to support all operating system environments. It won’t take much time for testing This will take a lot of time in once automation is done. running and maintaining and re- running. An automation test tool is always An automated tool requires new easy to learn and use. skills; therefore, additional training is required. 100% test coverage can be Test coverage breadth and depth achieved. can be increased but 100% exhaustive testing cannot be done.
- 13. Automation Myths/Realities All possible Input combinations can There are instances where it will be tested using automation take years to test all input combinations. 1 All possible paths can be traversed Research shows there are using automation. instances when possible paths are in Millions and more. 2 Automation is just record/capture Is it? Not at all. Try changing id of and play back one control on the page and run your script. I won’t require a specialist for that It’s a special task, which is a mix of programming, knowledge of tool and understanding of application. Not every tester can do this.
- 14. Then what should I do to reap benefits from my automation efforts?
- 15. Let’s begin !!! • Identify the interfaces our tooling need to interface with. For instance, are we going to drive a browser, are we send webservices requests and validate the responses, are we validating email notifications, database validations, or document or image validations etc. • Did the QA or DEV develop any tool/libraries that we can use? • What is the focus on? Is it automating new functionality tests (a challenge with agile development) or catching up with regression tests (when there are several years of development with very little focus on automating). • Which test case tracking system our automated tests need to be integrated with. • What is the triggering mechanism for these tests and when? • Who need to contribute to the test scripts development? QA, Dev or both?
- 16. The Initial Plan • People make mistakes when they try to build an elaborate solution that attempts to achieve all requirements at once. • People plan for results but do not plan for the ways to achieve this. It’s necessary to understand what you eventually want to achieve through automation, but to get there, plan to have your Test automation development as an iterative process. • Try to first figure out solving which problem would give you the biggest bang for your buck. E.g. Building the Build Acceptance tests and automatically running them with, at every check-in, or at code deployment on a QA system, ends up being a good choice as the first goal. After that you can always build upon your successes.
- 17. Technology choices • Identify your goals and what resources you have to achieve them, then the technology choices become a lot simpler. For instance, if you intend to use APIs created by the development team or if they are directly contributing to the solution, then often the programming/ scripting language choice needs to be something developers are comfortable with. If you are working in a java shop, you can’t expect the developers to create VBScript libraries to be used in QTP. Several tools these days, like Selenium and Fitnesse support multiple language choices. • Once you narrow down on the tools that fit the requirements, the next steps are to start evaluating and prototyping them using a few simple test cases.
- 18. Which tool should I use? Commercial solutions Pros Cons Published feature road maps Vendor lock-in Institutionalized support Lack of interoperability with other products Stability (whether real Lack of control over improvements or perceived) Licensing costs and restrictions
- 19. Which tool should I use? Open Source solutions Pros Cons No licensing fees, maintenance, or The downsides of commercial restrictions tools might be applied to some open-source projects, but the Free and efficient support (though varied) advantages of leveraging the open-source community and its Platform portability efforts are holding sway with Modifiable and adaptable to suit your more and more companies. needs Comparatively lightweight Not tied to a single vendor
- 20. Which tool should I use? In-House solutions Pros Cons No licensing fees, maintenance, or This involves a complete product restrictions development life cycle leading to time and resources requirement. Free and efficient support The financial aspects associated with (though varied) its development makes organization skeptical of its use. Platform portability Maintenance is costly as it need to be adjusted with changing requirements. Modifiable and adaptable to suit Management needs to be patient with Your needs it. Comparatively lightweight
- 21. Criteria in Finding & Evaluating tools Simple installation/un- Comprehensive and complete installation of the product documentation. Regularly maintained with code changes Configurability—the ability to Strong Support Team, 24x7 be adapted to each evaluation availability and online activity 1 forums/user groups Expandability – The tool Tool should be extensible and should support various flexible to deal with new objects infrastructures and apps 2 in the new architectures developed as part of revisions. 3
- 22. Using Coding Standards in Automated Software Testing • Automated software testing efforts can fail if the software development doesn't take into account the automated testing technologies or framework in place. • There should be provisions kept for automation and resources should be assigned to it • Changes made in application code will force changes in automated software testing scripts and hence developers can contribute to the success of automated testing efforts if they consider the impacts on them when making code or technology changes.
- 23. Using Coding Standards in Automated Software Testing - Guidelines • Build testability into the application – Make your application testable. • Design to facilitate automation tool recognition of objects: All objects should be uniquely named, consider various platforms—client/server, Web, etc.—and GUI/interface testing considerations, such as in the case of Windows development, for example, within the Windows architecture. • Don't change the object names without automated software testing considerations. • Follow standard development practices; for example, maintain a consistent tab sequence. • Follow techniques, such as the library concept of code reuse, i.e., reusing existing already tested components, as applicable.
- 24. Thank You! Vipin Jain Sr. Software Test Lead, Metacube software, India [email protected]
Editor's Notes
- #3: There are various definitions given for software testing by various authors. This definition has been taken from Wikipedia.
- #4: (1) More bugs are found via Ad-hoc testing than via automation
- #7: 1. Reliable: Tests perform precisely the same operations each time they are run, thereby eliminating human error.Reduce manual software testing operations and eliminate redundant testing efforts.2. Repeatable: You can test how the software reacts under repeated execution of the same operations. 3.Programmable: You can program sophisticated tests that bring out hidden information from the application. 4. Comprehensive: You can build a suite of tests that covers every feature in your application. 5. Reusable: You can reuse tests on different versions of an application, even if the user interface changes. 6. Better Quality Software: Because you can run more tests in less time with fewer resources 7. Fast: Automated Tools run tests significantly faster than human users. 8. Cost Reduction: Automation can help to detect defects early in the QA cycle, saving a lot of cost and effort early on. As the number of resources for regression test are reduced cost reduces.
- #14: the test of a function that handles the verification of a user password. Each user on a computer system has a six to eight character long password, where each character is an uppercase letter or a digit. Each password must contain at least one digit. According to Kenneth H. Rosen in Discrete Mathematics and Its Applications, there are 2,684,483,063,360 possible variations of passwords. Even if it were possible to create a test procedure each minute, or 60 test procedures per hour, equaling 480 test procedures per day, it would still take 155 years to prepare and execute a complete test.The format of telephone numbers in North America is specified by a numbering plan. A telephone number consists of ten digits, which are split into a three-digit area code, a three-digit office code, and a four-digit station code. Because of signaling considerations, there are certain restrictions on some of these digits. A quick calculation shows that in this example 6,400,000,000 different numbers are available—and this is only the valid numbers; the number will be huge if you consider the invalid numbers as well.
- #16: As far as developers demoted to "test development", I think it can backfire, especially if the test developer does NOT have the mindset of what they're testing for. Let's face it! Testing is NOT a mindless task. It takes thought and the right mindset to come up with a good group of test cases. The only suggestion I can come up with for this case is to have a happy in between: Have the test architect build a framework that is intuitive enough that a test engineer can quickly build scripts from it. Of course, all this TAKES work and the automation framework requires time and effort to maintain.
- #18: Often people want to talk about tooling when they want to start automating but only once you understand what your goals are and what resources you have to achieve them, the technology choices become a lot simpler. For instance, if you intend to use APIs created by the development team or if they are directly contributing to the solution, then often the programming/ scripting language choice needs to be something developers are comfortable with. If you are working in a java shop, you can’t expect the developers to create VBScript libraries to be used in QTP. Several tools these days, like Selenium and Fitnesse support multiple language choices.
- #22: 1. Configurability—the ability to be adapted to each evaluation activity; refers to how easy it is to set up each new evaluation project 2. Expandability—whether the tool suite works on various applications and infrastructures 3. Extensibility and technology lag—all commercial tools will eventually experience a technology lag behind the architectures they are targeted to support. When new development architectures are revised to add new features for software developers, there is a good chance that test automation tools may not recognize new objects. Therefore, it is important to evaluate tool extensibility and/or flexibility when dealing with new objects.