Tool-Support-For-Testing-Section-6.pptx
- 1. Tool Support For Testing - Section 6 ISTQB Foundation Rogerio da Silva ISTQB Certified Member of BCS
- 2. Intro What you will learn in this sixth section • Test Tools • Benefits of Test Tools • Risks of Test Tools • Tools For Static Testing • Tools For Dynamic (Test Specifications) Testing • Tools for Execution and Logging • Tools for Performance and Monitoring • Introducing Test Tool Into an Organisation
- 3. Types of Test Tools Test automation or Computer-Aided Software Testing (CAST) is widely used in test organisations to support the test process. Test tools can be used for a number of activities that can support testing, including: • Tools used directly in testing tasks i.e. test execution, data generation, result comparison • Tools that help to manage the test process i.e. management of tests, requirements, data, incidents • Tools used to monitor and report activities during testing i.e. performance, memory, file activity • Generic tools that can be used to help testing i.e. spreadsheets, SQL, debugging tools
- 4. Types of Test Tools Testers may use tool support for a number of reasons, depending on the context: • Improve the efficiency of testing i.e. scripted test execution • Support manual activities i.e. test planning, design or reporting • Automate repetitive tasks i.e. regression testing • Automate activities that are difficult to do manually i.e. static analysis or large scale performance testing • Increase reliability of testing i.e. automating large data comparisons or simulating behaviour
- 5. Test Tools Classification There are a number of tools that support different aspects of testing. Tools can be classified based on several criteria such as purpose, commercial / free / open-source / shareware, technology used and so forth. Tools are classified in the ISTQB® Foundation syllabus according to the testing activities that they support. Some tools clearly support one activity; others may support more than one activity, but are classified under the activity with which they are most closely associated. For example, a test execution tool may incorporate a built-in test comparator, but it is classified as a test execution tool rather than a comparator tool. One of the most important features of tools is their ability to interact and integrate with other tools. Tools from a single provider, especially those that have been designed to work together, may be bundled into one package. Some tools offer support that is more appropriate for developers than testers (i.e. tools that are used during component and component integration testing). Such tools are marked with (D) in the list as follows.
- 6. Tools Support for Management of Testing and Tests Management tools apply to all test activities over the entire software life cycle.
- 7. Test Management Tools These are ‘umbrella’ tools which provide interfaces for executing tests, tracking defects and managing requirements. They also support traceability of test objects to requirements specifications, and might have an independent version control capability or an interface to an external configuration management tool. They typically support quantitative analysis and reporting of test activity in a wide variety of formats. Test Management Tool Requirement Management Tool Configuration Management Tool Incident Management Tool Test Execution Tool
- 8. Requirements Management Tools These tools store business requirements in a central repository which incorporates • unique identifiers • descriptions of functional and non-functional requirements • attributes such as source, priority, rationale and status • links between requirements These tools may also help with identifying inconsistent, overlapping or missing requirements, and can support tracing the requirements to individual tests. ARM (Automated Requirement Measurement) is a free tool provided by the NASA Goddard Space Flight Centre to scan requirements specifications for specific words and phrases indicative of quality problems, such as weak, fuzzy,
- 9. Incident Management (Defect Tracking) Tools These tools store and manage incident reports, such as defects, failures, change requests or perceived problems and anomalies, and manage the life cycle of incidents, optionally with support for statistical analysis. They support the logging of incident characteristics (i.e. type, priority, severity) and status (i.e. rejected, ready, deferred). These may be bundled with other test management tools or available separately, including some free tools. Many tools can produce graphs based on different parameters from the mass of incident data that is typically logged during testing. There are several tools out there, the most common ones are: Jira, Mantis, Bugzilla, HP ALM Quality Centre
- 10. Configuration Management (CM) Tools Although not strictly test tools, these can support the test process. They are necessary for storage and version management of test ware and related software especially when configuring more than one hardware/software environment in terms of operating system versions, compilers, browsers, etc. They store version/build/configuration information, support tracking and control of testing and development products, and provide direct integration with many testing tools. CM tools may have to cater for many platforms and artefacts across a large organisation, so a specialist tools is likely to be better than on the is bundled with a CASE or test tool. There are several tools out there, the most common ones are:
- 11. Tools Support for Static Testing Static testing tools provide a cost effective way of finding more defects at an earlier stage in the development process. There are several tools out there, the most common ones are: Eclipse, IntelliJ IDEA, Visual Studio
- 12. Review Tools These tools (also known as Review Process Support tools) assist with review processes, checklists, review guidelines and defects. They are used to store and communicate reviewers’ comments, report on defects and effort, and store metrics for causal analysis and process improvement. They can be of further help by providing aid for online reviews for large or geographically dispersed teams. Review tools are most useful for formal inspections which utilise databases, forms, checklists and causal analysis. There are several tools out there, the most common ones are: IntelliJ IDEA, FindBugs, Checkstyle, jarchitect
- 13. Static Analysis Tools (D) These tools help developers and testers find defects in source code prior to dynamic testing. Static analysis tools can provide complexity feedback for developers via various industry-defined metrics. They can analyse structures and dependences and help in planning and risk analysis. Subtle coding errors and fault-prone code can often be found far more quickly using these tools than by visual inspection or testing. Code may also be reviewed and sorted using user-defined criteria including metrics, often generating automatic reports and blocking code when a threshold is exceeded until the problem is resolved. Static analyses may also be customised to check that organisational standards are maintained as well as industry-wide measures. Static analysers should allows messages filtering otherwise the output can be overwhelming. There are several tools out there, the most common ones are: IntelliJ IDEA, Eclipse, Visual Studio
- 14. Modelling Tools (D) These tools are used to validate software models (i.e. physical data model, activity model, process model), by enumerating inconsistencies and finding defects. These tools can often aid in generating some test cases based on the model. Although mainly used by designers and developers, modelling tools that record state transition diagrams or use cases can also be suitable for testers. Even if they don’t auto-generate tests, the way that sequence or state diagrams show transitions is ideally suited to deriving tests. Similarly a use case that describes pre and post conditions and a series of steps simplifies the test design process. There are several tools out there, the most common ones are: Microsoft Visio, MySQL Workbench, NetBeans, Eclipse UML2 Tools
- 15. Tool for Test Specification – Test Design Tools Test Design Tool Requirements GUI Design Models (state, data or object) Source code Test inputs Executable tests Test oracles (expected results) These tools are used to generate executable tests, test inputs (data) or test oracles (expected results) automatically from requirements, graphical user interfaces, design models (such as business rules, state diagrams or data models) or code. Code-based test generators are similar to static analysers and are language-specific. Typically they identify insertion points for exception testing or identify paths and path conditions, especially useful when there is undocumented software. The generated tests from a state or object model are useful for verifying the implementation of the model in the software, but are seldom sufficient for verifying all aspects of the software or system. Test Design tools can save valuable time and provide increased thoroughness of testing because of the completeness of the tests that the tool can generate.
- 16. Tool for Test Specification – Data Preparation Tools Test data preparation tools manipulate databases, files or data transmissions to set up test data to be used during the execution of tests. Test data preparation tools enable data to be selected from existing databases or created afresh. If tests are using copies of production data with sensitive information, these tools can ensure security through data anonymity. Large-scale random data can be generated according to defined business rules. Base data records will need to be set up in a certain state which will enable test cases to work. This may require a complex series of transactions to achieve, but can more easily achieved by artificial means. There are several tools out there, the most common ones are: MySQL Workbench, GSApps, Datatect
- 17. Tool for Test Execution and Logging – Test Execution Tools These tools enable tests to be executed automatically, or semi-automatically, using stored inputs and expected outcomes, through the use of a scripting language. They can also be used to record tests, and usually support scripting language or GUI-based configuration or parameterisation of data and other customisation in the tests. Test execution tools usually provide a test log for each test run and can compare actual and expected results. A GUI capture/playback tool forms the heart of most execution tools. Keystrokes and mouse pointer action are intercepted by the tool and stored in a test script which will enable replay. The state of GUI objects such as windows, fields, buttons and other controls as well as display output is also recognised and recorded. Scripts can then be repeated or amended to vary the tests run. Data, test cases and expected results may be held in separate test repositories and varied as required. There are several tools out there, the most common ones are:
- 18. Tool for Test Execution and Logging – Test Harness / Unit Test Frameworks Tools (D) A unit test harness or framework facilitates the testing of components or parts of a system by simulating the environment in which that test object will run, through the provision of mock objects as stubs or drivers. This is particularly useful for component integration testing, such as top-down or bottom-up testing. Test harnesses and drivers are used to execute software under test which may not have a user interface or to run groups of existing automated test scripts which can be controlled by the tester. Stubs are used to simulate routines which have not yet been written, and can print or display the values passed to them. Drivers are used to pass variables to a routine under test, and print or display variables returned from them. Simulators are used to support test where code or other systems are either unavailable or impracticable to use (i.e. testing software to cope with nuclear meltdowns). There are several tools out there, the most common ones are: Junit (Java), Nunit (.Net), QTP, HP ALM, Selenium, Eclipse, IntelliJ IDEA
- 19. Tool for Test Execution and Logging – Test Comparator Tools Test comparator tools are used to compare actual results and expected results by analysing differences between files, databases or test results. Standalone comparison tools normally deal with a range of file or database formats. But many test execution tools have built-in comparators that deal with character screens, GUI objects or bitmap images. These tools often have filtering or masking capabilities, so they can ignore rows or columns of data, or areas screens. Comparator tools which check printed output are more specialised, as they need to avoid throwing up differences on variable print data like page numbers, line numbers, run dates and times. They do need to check whether the same messages and values are present on the print files for the same identifier (i.e. master file record code). There are several tools out there, the most common ones are: Ultracompare, WinMerge, DiffDoc and Windiff
- 20. Tool for Test Execution and Logging – Coverage Measurement Tools These tools measure the percentage of specific types of code structures that have been exercised by a set of dynamic tests. They may measure coverage of statements, branches or decisions, and calls to modules or functions. They are used during white-box testing to measure actual coverage achieved by a set of structural tests. Such tools may be intrusive – i.e. they add extra code to the software under test to measure the coverage. This is known as the probe effect. Coverage measurement tools typically produce detailed coverage reports or graphs showing what code has been executed, the number of times particular statements or decision outcomes have been executed, and non-executable statements such as declarations and comments. Error handlers can be specifically included or excluded from the reported results and results can be tailored to include all test runs or just the current run. It may be possible to view and monitor coverage increasing on-line while the software is running, and results may be exported to a spreadsheet for further analysis. There are several tools out there, the most common ones are:
- 21. Tool for Test Execution and Logging – Security Tools These tools are used to evaluate security characteristics of software such as the ability of the software to protect data confidentially, integrity, authentication, authorisation, availability and non-repudiation. They can also check for viruses and denial-of-service attacks. Due to the nature of these tools, they are mostly focused on a particular technology, platform and purpose. These are essential in order to protect confidential business systems and sensitive data from attacks on web-based or distributed systems. They are often very sophisticated tools and are more likely to be used by a security specialist than by a typical tester. There are several tools out there, the most common ones are: SQLMap, Google Nogotofall, Vega, ZED Attack Proxy (ZAP)
- 22. Tool for Performance and Monitoring – Dynamic Analysis Tools (D) Dynamic analysis tools provide run-time information on the state of software under test. They find defects that are evident only when software is executing, such as time dependencies or memory leaks. For example, the memory on a workstation may continuously decline leading ultimately to an untidy application failure. They are typically used by developers in component and component integration testing, and when testing middleware (application software that connects software components).
- 23. Performance, Load and Stress Testing Tools Performance testing tools monitor and report on how a system behaves under a variety of simulated usage conditions in terms of number of concurrent users, their ramp-up pattern, frequency and relative percentage of transactions. The purpose of performance testing is to show that the system has the capacity to handle large number of users and transactions during normal operation and peal periods to comply with any Service Level Agreement on response times. Load testing assesses the capability of the infrastructure to handle the expected loads, while stress testing evaluates the system beyond its expected limits. The simulation of load is achieved by means of creating virtual users carrying out a selected set of transactions, spread across various test machines commonly known as load generators. Performance testing tools normally provide reports based on test logs, and graphs of load against response times. Larger companies may have specialist performance test teams to design tests and analyse the results. There are several tools out there, the most common ones are: SQLMap, Google Nogotofall, Vega, ZED Attack Proxy (ZAP)
- 24. Tool for Performance and Monitoring – Monitoring Tools Monitoring tools continuously analyse, verify and report on usage of specific system resources (such as CPU, memory, disc capacity, network), and give warnings of possible service problems. These are really part of systems Management tools rather than being specialised tester tools, but can be used during dynamic testing to highlight system resources issues which might have an impact on availability, performance or other SLAs. There are several tools out there, the most common ones are: New Relic, WinFeedback, AppsWatch and nGenius Performance Monitor
- 25. Tool for Performance and Monitoring – Data Quality Assessment Tools These tools assess data quality and integrity for data-centric projects such as data conversion/migration projects and data warehouse applications. They can review and verify data conversion and migration rules, and ensure that processed data is correct, complete, accessible and compliant with pre-defined standards. There are several tools out there, the most common ones are: Informatica Data Validation, QuerySurge from RTTS, iCEDQ and Datagaps ETL Validator
- 26. Effective Use of Tools Simply purchasing or leasing a tool does not guarantee success with that tool. Each type of tool may require additional effort to achieve real and lasting benefits. There are potential benefits and opportunities with the use of tools in testing, but there are also risks.
- 27. Effective Use of Tools – Potential Benefits Possible Benefits of using tools are: • Reduce repetitive work, i.e. regression testing, re-entering the same test data, checking against coding standards • Ensure greater consistency and repeatability, i.e. tests executed in the same order with the same frequency • Improve accuracy by removing human error • Provide objective assessment and reporting of static measures, coverage and system behaviour • Maintain history of test documents i.e. test plans, specification and logs • Provide easy access to information about testing, i.e. statistics and graphs about test progress, incident rates
- 28. Effective Use of Tools – Potential Risks Risks of using tools are: • Unrealistic expectations for the tool, including functionality and easy of use • Underestimating the costs, time and effort required to introduce a tool and to achieve significant and continuing benefits, including training, the need for changes in the testing process and continuous improvement. • Over-reliance on the tool, such as using automated testing when manual test testing would be better. • Neglecting version control of test assets within the tool • Neglecting relationships and interoperability issues between tools, such as requirements management tools, and tools from multiple vendors • Tool supplier issues such as poor response or support, or supplier going out of business • Suspension of open-source or free tools • Inability to support a new platform
- 29. Special Consideration for Some Type of Tools Some tools have particular issues which need careful consideration before being implemented within a testing organisation.
- 30. Static Analysis Tools These tools execute tests using automated test scripts, generated by recording the actions of a manual testers using capture-replay technology. This type of tool often requires significant effort in order to achieve significant benefits. Such scripts contain both the actions and data from the recorded test, which limits their flexibility. To obtain the greatest benefit, it is necessary to de-couple the actions and data, which can then be modified to create a more flexible set of potential tests. This process, and the on-going maintenance of automated scripts, requires technical expertise in the scripting language, either by testers or by specialists in test automation.
- 31. Static Analysis Tools A data-driven testing approach separates out the test inputs (data), usually into a spreadsheet, which can then be edited without knowledge of a scripting language. The generic test script, containing the original actions, can then be run many times against different data in the spreadsheet, or against data generated using algorithms based on configurable parameters at run time. vs A keyword-driven testing approach separates out the actions, in the form of standard keywords, into a spreadsheet. These can then be modified by testers (even if they are not familiar with the scripting language) to define new tests, tailored to the application being tested.
- 32. Test Management Tools Test management tools need to interface with other tools (such as requirements management and defect management tools) or spreadsheets in order to produce useful information in a format that fits the needs of the organisation. It is therefore essential to ensure that these tools can communicate successfully, particularly if they are from different suppliers.
- 33. Introducing a Tool Selection Considerations The introduction of a testing tool is costly, so it must be treated like any other business project, with clear business objectives and an estimate of cost-benefit ratio based on a concrete business case to justify the expenditure. Before introducing a tool into an organisation, it is important to assess the testing maturity, strengths and weaknesses of the testing process and team, and to identify opportunities for process improvement using tools. Any prospective tool must be evaluated against clear requirements and objective criteria. A feasibility study (proof-of-concept) can be established to demonstrate that a tool performs effectively with the software under test and with the current infrastructure, and to identify any changes needed for effective use of the tool. The tool supplier must be evaluated to clarify what provision they make for training, service support, upgrades and commercial aspects. The test team’s requirements for coaching and mentoring should also be considered, especially the team’s test automation skills.
- 34. Introducing the Selected Tool Before making a commitment to implementing a tool within an organisation, a pilot project is usually undertaken to ensure the benefits of using the tool can actually be achieved. The objectives of the pilot are to learn more detail about the tool, evaluate how it fits with existing processes and practices, identify changes required in the test process and assess whether the benefits will be achieved at reasonable cost. It is also an opportunity to decide on standard ways of using, managing, storing and maintaining the tool and the test assets (i.e. naming conventions for files and tests, creating libraries, defining modularity of test suites and common conventions for use.)
- 35. Deployment Success Factors Full deployment of the tool should be based on a successful result from the evaluation of the pilot. This should be assessed along with implementation plan to ensure it is right to proceed and that everything is ready for a successful roll-out. Success factors for deployment of a tool within an organisation include: • Roll out incrementally across the organisation • Adapt and improve processes to fit with use • Provide training and coaching/mentoring for new users • Define usage guidelines • Implement a way to gather usage information • Monitor use and benefits • Provide support for the test team for a given tool • Gather lessons learnt from all teams
- 36. Summary This is what you have learned so far... • Test Tool • Benefits Of Test Tools • Risk Of Test Tools • List Of Tools For Management Of Testing and Test • List Of Tools For Static Testing • List Of Tools For Test Specification (Dynamic) • List Of Tools For Testing Execution and Logging • List Of Tools For Performance And Monitoring • Steps For Introducing A Tool Into An Organisation
- 37. End of Section 6 Do the Quiz (attached) for Section 6 Questions? Comments? Thumbs up and Share it if you like this. Tool Support For Testing - Section 6 ISTQB Foundation Thank You