13090016_vectorcast.ppt
- 1. Logo of your organization 1 A scalable approach for Test Automation in Vector CAST/Manage with the Jenkins tool to achieve Requirement Based Testing. Manoj kumar Pant -SeniorEngineer Avneet Mishra- Senior Engineer Mando Softtech India
- 2. 2 Content • Introduction • Automotive Software Development and ISO – 26262 • What is VectorCAST/Manage ? • What is Jenkins? • VectorCAST/Manage integration with Jenkins provides a test automation platform through continuous Integration testing . • Benefits of continuous Integration testing . • Features provided by VectorCAST/Manage with Jenkins • Types of Requirement based testing performed by VectorCAST/Manage • Design methods of for Deriving test cases in VectorCAST/Manage for Software Unit/integration Testing. • Coverage analysis through VectorCAST/Manage
- 3. Introduction • This paper explain requirement based testing approach for ensuring the intended Automotive Software performing its functionality with a high level of confidence by satisfying the ISO 26262 functional safety standard to meeting the Aspice process assessment objectives by using VectorCAST/Manage and Jenkins tool together. • The objective of this document is to standardize the test case design and test procedure development by using the VectorCAST/Manage and Jenkins framework for the Unit level testing, Integration Testing and Hardware Software Integration Testing (Known as System Testing) as per ISO 26262. 3
- 4. Vector CAST/Manage integration with Jenkins tool to achieve Requirement Based Testing 4
- 5. 5 Automotive Software Development and ISO – 26262:Automotive industries follow the V - model for software development life cycle to meeting the objectives of ISO 26262 for software safety standard compliance and quality as shown. V-model of software development life cycle with reference to ISO 26262
- 6. What is VectorCAST/Manage ? • VectorCAST/Manage is a tool for automating the process associated with different levels of testing such as Unit level, Integration level and System level as shown. It is used for code-coverage analysis by identifying which areas of an application have been exercised by test executions. 6 Vector CAST/Manage tool to achieve Requirement Based testing
- 7. What is Jenkins ? Jenkins is the leading open-source continuous integration server. It uses the command to compile that stimulates a complete simulation environment for an automotive system testing. Jenkins enables continuous testing each time a source code change is made. it provides a distributed testing infrastructure which allows a list of ‘nodes’ .nodes indicates the types of job that can be run and reports on job status when complete. 7
- 8. VectorCAST/Manage integration with Jenkins provides a test automation platform through continuous Integration testing : • Continuous integration (CI) testing ensure software to be developed rapidly, reliably, and repeatedly with minimal human intervention. • Most automotive Industry (OEM and supplier) focusing on Continuous Integration (CI) process to reduce build times. • VectorCAST/Manage Integration with Jenkins provides a platform for continuous Integration testing by Creating Single jobs, Multiple Jobs and Diagnostic Job. • A Continuous Integration workflow approach provides incremental and parallel source code compilation resulting in significant reduced application build times. • The Continuous Integration allow compile the changed files only, and those compilation jobs are spread over multiple cores on a single server, or multiple physical servers. • Continuous Testing applies these same techniques for software testing, thus reducing test cycle times by selecting the sub-set of tests that are affected by a set of code changes, and thus running those tests in parallel. 8
- 9. Benefits of Continuous Integration testing: VectorCAST/Manage with Jenkins provides the Continuous Testing (CI) to identify a defect (majorly at integration level) and provide instant feedback so that it can be detected as soon as possible. Using automated tests, locating and fixing of bugs becomes easy because any bug in the process is quickly identified by relating it to the commit. This increases the frequency of software releases, time to market and ensures more reliable applications. . 9 VectorCAST/Manage automation increase the speed of testing by reducing time factor
- 10. 10 VectorCAST/Manage work together with Jenkins to reduce the time factor: Let’s assume that each test suite contains 10,000 individual test cases, each of which takes 5 minutes to run. if we perform testing without Jenkins, the testing would be done in 50,000 minutes or 5 weeks. By configuring multiple “nodes” to execute tests in Jenkins, we can divided the testing work across many machines, and shorten the total time to test. Taken to an extreme, we could have 10,000 Jenkins Slave virtual machines running, and our testing would be completed in 5 minutes. This approach Speeds overall project testing time and reduces late-in-the-project side effects. VectorCAST/Manage with Jenkins reduces testing time with a huge margin
- 11. . 11 VectorCAST/Manage with Jenkins provides the following features: • Ability to organize thousands of test cases into logical groups that conform to the structure of your application. • Ability to run the same test against multiple configurations of the application. • Ability to determine the pass/fail status of each test case. • Ability to determine code coverage per function Allows tests to be easily run on every source change. • Ability to define a list of “nodes” which are physical or virtual machines. • Ability to e-mail job status once jobs are complete. • Perfect for large projects with many users and a lot of tests. • Overnight or complete application test execution can be reduced from days to hours. • Impact of Change analysis (Regression testing) can be performed on the master project, greatly reducing the time it takes to identify regression errors.
- 12. Types of Requirement based testing performed by VectorCAST/Manage : VectorCAST/Manage during Hardware/software integration testing: To verify correct operation of the software in the target computer environment. VectorCAST/Manage during Software integration testing: To verify the interrelationships between software requirements and components and to verify the implementation of the software requirements and software components within the software architecture. VectorCAST/Manage during Low-level testing (Unit Testing): To verify the implementation of software low-level requirements. 12 SOFTWARE TESTING ACTIVITIES
- 13. VectorCAST/Manage during Software Unit /integration Testing: Requirements-Based Test: Requirement-based test is a method where a software/System requirement, such as functionality is verified. The generation of requirement-based tests is greatly facilitated within VectorCAST/Manage through the use of a graphical user interface (GUI) where the input and expected values of parameters, return values, global variables and even stubs (a piece of code that replaces dependencies) can be specified. Interface Test: This can be done by using boundary values of the interface, illegal values and median values. The main objective is to ensure that the interface is robust so as to prevent errors from occurring. 13
- 14. 14 Fault injection Test: : ISO26262 uses fault injection tests to increase the test coverage of safety requirements as shown :
- 15. • Resource Usage Test: Resource usage testing can be automated by using the VectorCAST/Manage tool. • Back-to-Back Test between Model and Code: VectorCAST/Manage provides integrations with design and modelling tools causing automatic code generation, which can be input directly into VectorCAST for testing. Design Methods for Deriving test cases in VectorCAST/Manage for Software Unit/integration Testing: 15 ASIL A B C D 1a Analysis of requirements ++ ++ ++ ++ 1b Generation and analysis of equivalence classes + ++ ++ ++ 1c Analysis of boundary values + ++ ++ ++ 1d Error guessing + + + +
- 16. 16 • Analysis of requirements:Requirement analysis is done from Requirement specification document such as Software Requirement Specification (SRS) and Software Design document (SDD). • Test case sample for requirement based testing:
- 17. 17 Requirement based testing can be designed in VectorCAST/Manage by following methods: Normal Range Test case design: To verify the normal behaviour of intended software test case should be designed: (1)within the data range (2)Using valid equivalence class and boundary values Robustness Test case design: To verify the abnormal behaviour of intended software test case should be designed: (1)Outside the data range. (2) Using Invalid equivalence class. Example : If the Range of parameter is 1 to 200. Normal range test case should be developed from 1 to 200. Abnormal range test case should be developed from <1 and >200.
- 18. Below are the following Requirement based testing techniques performed through VectorCAST/Manage: 1) Equivalence classes 2) Boundary Value Analysis Generation and analysis of equivalence classes in VectorCAST/Manage: In this method, the data is divided into set of valid and invalid class in vector cast GUI. Example : If the Range of parameter is 1 to 200. 18 Equivalence Class Partitioning in VectorCAST/Manage tool
- 19. Analysis of boundary values in VectorCAST/Manage: Boundary value analysis’ testing technique is used to identify errors at boundaries rather than finding those exist in centre of input domain. Example : If the Range of parameter is 1 to 200. 19 Boundary Value Analysis in VectorCAST/Manage
- 20. Error Guessing Test Error guessing is a testing technique that makes use of a tester's skill, intuition and experience in testing similar applications to identify defects that may not be easy to capture by more formal techniques. It is usually done after most of the formal techniques are completed. 20
- 21. Coverage analysis through VectorCAST/Manage • The ISO 26262 divides ASIL levels as follows: • ASIL A • ASIL B • ASIL C • ASIL D • Functional safety standard of ISO 26262, Part 6, Section 9.4.5, Table-12 which is shown in figure 14, • ASIL levels are used as per the criticality of the software. The ASIL A level only covers statements, ASIL B/C level covers statement and branch and ASIL level D covers statements, branch and MCDC (Modified Condition Decision control). 21
- 22. 22 Method ASIL Level A B C D Statement Coverage ++ ++ + + Branch Coverage + ++ ++ ++ MC/DC Coverage + + + ++ Automotive Safety Integrity Levels
- 23. Coverage analysis through VectorCAST/Manage 23
- 24. Coverage trend through VectorCAST/Manage 24
- 26. 26 Thank You!!! Logo of your organization