Hai
- 1. Software Testing Principles Principles play an important role in all engineering disciplines and are usually introduced as part of an educational background in each branch of engineering. Testing principles are important to test specialists/ engineers because they provide the foundation for developing testing knowledge and acquiring testing skills. They also provide guidance for defining testing activities as performed in the practice of a test specialist. A principle can be defined as: 1. a general or fundamental, law, doctrine, or assumption; 2. a rule or code of conduct; 3. the laws or facts of nature underlying the working of an artificial device. In the software domain, principles may also refer to rules or codes of conduct relating to professionals who design, develop, test, and maintain software systems. Testing as a component of the software engineering discipline also has a specific set of principles that serve as guidelines for the tester. They guide testers in defining how to test software systems, and provide rules of conduct for testers as professionals. Principle 1. Testing is the process of exercising a software component using a selected set of test cases, with the intent of (i) revealing defects, and (ii) evaluating quality Software engineers have made great progress in developing methods to prevent and eliminate defects. However, defects do occur, and they have a negative impact on software quality. Testers need to detect these defects before the software becomes operational. This principle supports testing as an execution-based activity to detect defects. It also supports the separation of testing from debugging since the intent of the latter is to locate defects and repair the software. The term “software component” is used in this context to represent any unit of software ranging in size and complexity from an individual procedure or method, to an entire software system. The term “defects” as used in this and in subsequent principles represents any deviations in the software that have a negative impact on its functionality, performance, reliability, security, and/or any other of its specified quality attributes. Bertolino, in the Guide to the Software Engineering Body of Knowledge, gives a view of testing as a „„dynamic process that executes a program on valued inputs‟‟,to detecting defects, testing is also a process used to evaluate software quality. The purpose of the former has been described in the previous paragraph.
- 2. In the case of the latter, the tester executes the software using test cases to evaluate properties such as reliability, usability, maintainability, and level of performance. Test results are used to compare the actual properties of the software to those specified in the requirements document as quality goals. Deviations or failure to achieve quality goals must be addressed. Principle 2. When the test objective is to detect defects, then a good test case is one that has a high probability of revealing a yet undetected defect(s). Principle 2 supports careful test design and provides a criterion with which to evaluate test case design and the effectiveness of the testing effort when the objective is to detect defects. It requires the tester to consider the goal for each test case, that is, which specific type of defect is to be detected by the test case. In this way the tester approaches testing in the same way a scientist approaches an experiment. In the case of the scientist there is a hypothesis involved that he/she wants to prove or disprove by means of the experiment. In the case of the tester, the hypothesis is related to the suspected occurrence of specific types of defects. The goal for the test is to prove/disprove the hypothesis, that is, determine if the specific defect is present/absent. Based on the hypothesis, test inputs are selected, correct outputs are determined, and the test is run. Results are analyzed to prove/disprove the hypothesis. The reader should realize that many resources are invested in a test, resources for designing the test cases, running the tests, and recording and analyzing results. A tester can justify the expenditure of the resources by careful test design so that principle 2 is supported. Principle 3. Test results should be inspected meticulously Testers need to carefully inspect and interpret test results. Several erroneous and costly scenarios may occur if care is not taken. For example: • A failure may be overlooked, and the test may be granted a “pass” status when in reality the software has failed the test. Testing may continue based on erroneous test results. The defect may be revealed at some later stage of testing, but in that case it may be more costly and difficult to locate and repair.
- 3. • A failure may be suspected when in reality none exists. In this case the test may be granted a “fail” status. Much time and effort may be spent on trying to find the defect that does not exist. A careful reexamination of the test results could finally indicate that no failure has occurred. • The outcome of a quality test may be misunderstood, resulting in unnecessary rework, or oversight of a critical problem. Principle 4. A test case must contain the expected output or result. It is often obvious to the novice tester that test inputs must be part of a test case. However, the test case is of no value unless there is an explicit statement of the expected outputs or results, for example, a specific variable value must be observed or a certain panel button that must light up. Expected outputs allow the tester to determine (i) whether a defect has been revealed, and (ii) pass/fail status for the test. It is very important to have a correct statement of the output so that needless time is not spent due to misconceptions about the outcome of a test. The specification of test inputs and outputs should be part of test design activities. In the case of testing for quality evaluation, it is useful for quality goals to be expressed in quantitative terms in the requirements document if possible, so that testers are able to compare actual software attributes as determined by the tests with what was specified. Principle 5. Test cases should be developed for both valid and invalid input conditions. A tester must not assume that the software under test will always be provided with valid inputs. Inputs may be incorrect for several reasons. For example, software users may have misunderstandings, or lack information about the nature of the inputs. They often make typographical errors even when complete/correct information is available. Devices may also provide invalid inputs due to erroneous conditions and malfunctions. Use of test cases that are based on invalid inputs is very useful for revealing defects since they may exercise the code in unexpected ways and identify unexpected software behavior. Invalid inputs also help developers and testers evaluate the robustness of the software, that is, its ability to recover when unexpected events occur (in this case an erroneous input). Principle 5 supports the need for the independent test group called for in Principle 7 for the following reason. The developer of a software component may be biased in the selection of test inputs for the component and specify only valid inputs in the test cases to demonstrate that the software works correctly. An independent tester is more apt to select invalid inputs as well.
- 4. Principle 6. The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component. What this principle says is that the higher the number of defects already detected in a component, the more likely it is to have additional defects when it undergoes further testing. For example, if there are two components A and B, and testers have found 20 defects in A and 3 defects in B, then the probability of the existence of additional defects in A is higher than B. This empirical observation may be due to several causes. Defects often occur in clusters and often in code that has a high degree of complexity and is poorly designed. In the case of such components developers and testers need to decide whether to disregard the current version of the component and work on a redesign, or plan to expend additional testing resources on this component to insure it meets its requirements. This issue is especially important for components that implement mission or safety critical functions. Principle 7. Testing should be carried out by a group that is independent of the development group. This principle holds true for psychological as well as practical reasons. It is difficult for a developer to admit or conceive that software he/she has created and developed can be faulty. Testers must realize that (i) developers have a great deal of pride in their work, and (ii) on a practical level it may be difficult for them to conceptualize where defects could be found. Even when tests fail, developers often have difficulty in locating the defects since their mental model of the code may overshadow their view of code as it exists in actuality. They may also have misconceptions or misunderstandings concerning the requirements and specifications relating to the software. The requirement for an independent testing group can be interpreted by an organization in several ways. The testing group could be implemented as a completely separate functional entity in the organization. Alternatively, testers could be members of a Software Quality Assurance Group, or even be a specialized part of the development group, but in the latter case especially, they need the capability to be objective. Reporting to management that is separate from development can support their objectivity and independence. As a member of any of these groups, the principal duties and training of the testers should lie in testing rather than in development.
- 5. Finally, independence of the testing group does not call for an adversarial relationship between developers and testers. The testers should not play “gotcha” games with developers. The groups need to cooperate so that software of the highest quality is released to the customer. Principle 8. Tests must be repeatable and reusable. Principle 2 calls for a tester to view his/her work as similar to that of an experimental scientist. Principle 8 calls for experiments in the testing domain to require recording of the exact conditions of the test, any special events that occurred, equipment used, and a careful accounting of the results. This information is invaluable to the developers when the code is returned for debugging so that they can duplicate test conditions. It is also useful for tests that need to be repeated after defect repair. The repetition and reuse of tests is also necessary during regression test (the retesting of software that has been modified) in the case of a new release of the software. Scientists expect experiments to be repeatable by others, and testers should expect the same Principle 9. Testing should be planned. Test plans should be developed for each level of testing, and objectives for each level should be described in the associated plan. The objectives should be stated as quantitatively as possible. Plans, with their precisely specified objectives, are necessary to ensure that adequate time and resources are allocated for testing tasks, and that testing can be monitored and managed. Test planning activities should be carried out throughout the software life cycle (Principle 10). Test planning must be coordinated with project planning. The test manager and project manager must work together to coordinate activities. Testers cannot plan to test a component on a given date unless the developers have it available on that date. Test risks must be evaluated. For example, how probable are delays in delivery of software components, which components are likely to be complex and difficult to test, do the testers need extra training with new tools? A test plan template must be available to the test manager to guide development of the plan according to organizational policies and standards. Careful test planning avoids wasteful “throwaway” tests and unproductive and unplanned “test–patch–retest” cycles that often lead to poor-quality software and the inability to deliver software on time and within budget. Principle 10. Testing activities should be integrated into the software life cycle.
- 6. It is no longer feasible to postpone testing activities until after the code has been written. Test planning activities as supported by Principle 10, should be integrated into the software life cycle starting as early as in the requirements analysis phase, and continue on throughout the software life cycle in parallel with development activities. In addition to test planning, some other types of testing activities such as usability testing can also be carried out early in the life cycle by using prototypes. These activities can continue on until the software is delivered to the users. Organizations can use process models like the V-model or any others that support the integration of test activities into the software life cycle Principle 11. Testing is a creative and challenging task Difficulties and challenges for the tester include the following: • A tester needs to have comprehensive knowledge of the software engineering discipline. • A tester needs to have knowledge from both experience and education as to how software is specified, designed, and developed. • A tester needs to be able to manage many details. • A tester needs to have knowledge of fault types and where faults of a certain type might occur in code constructs. • A tester needs to reason like a scientist and propose hypotheses that relate to presence of specific types of defects. • A tester needs to have a good grasp of the problem domain of the software that he/she is testing. Familiarly with a domain may come from educational, training, and work-related experiences. • A tester needs to create and document test cases. To design the test cases the tester must select inputs often from a very wide domain. Those selected should have the highest probability of revealing a defect (Principle 2). Familiarly with the domain is essential. • A tester needs to design and record test procedures for running the tests. • A tester needs to plan for testing and allocate the proper resources. • A tester needs to execute the tests and is responsible for recording results. • A tester needs to analyze test results and decide on success or failure for a test. This involves understanding and keeping track of an enormous amount of detailed information. A tester may also be required to collect and analyze test-related measurements.
- 7. • A tester needs to learn to use tools and keep abreast of the newest test tool advances. • A tester needs to work and cooperate with requirements engineers, designers, and developers, and often must establish a working relationship with clients and users. • A tester needs to be educated and trained in this specialized area and often will be required to update his/her knowledge on a regular basis due to changing technologies