20141024 AgileDC 2014 Conf How much testing is enough for software that can condemn a man to death? Traceability in an Agile criminal justice context
1 like519 views
The document discusses the importance of requirements traceability within agile software development, particularly in federally regulated environments. It outlines the limitations of traditional testing practices and advocates for Behavior-Driven Development (BDD) and Acceptance Test-Driven Development (ATDD) as alternative approaches to enhance traceability and ensure quality. The presentation emphasizes the need for automated test frameworks like SpecFlow to maintain assurance in software functionality, especially for mission-critical and life-critical systems.
20141024 AgileDC 2014 Conf How much testing is enough for software that can condemn a man to death? Traceability in an Agile criminal justice context
- 1. 1 Traceability in an Agile Federal Agency Context Craeg Strong CTO, Ariel Partners October 21, 2014 Washington, DC How Much Tes,ng Is Enough… When it Comes to Ma8ers of Life and Death?
- 2. 2 So&ware Development since 1988 Large Commercial & Government Projects Turned Around Projects With Agile Apache Ant Open-‐Source Contributor New York & Washington DC Area CTO, Ariel Partners CSM, CSP, CSD, CSPO, PSM, PMI-‐ACP, PMP www.arielpartners.com [email protected] @ckstrong1 Craeg Strong © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 3. 3 Agenda 1. What Requirements Traceability Is 2. Why & When It Is Important 3. Some Typical Agile PracZces 4. Why They Are Insufficient 5. AlternaZve Approach: BDD/ATDD With SpecFlow 6. How/When/Who/Why 7. Solving The Problem With Agility 8. How To Get There: TransiZonal Strategies © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 4. 4 Context Examples Use Microsoft technology Equivalents Exist in Other Language Ecosystems (Java, LAMP) q Air traffic control q Loss reserve calculation for long-tail insurance q Threat evaluation for detainee at border crossing q Nuclear power plant control q Digitized forensic evidence match at murder trial q Trading risk circuit breaker for high frequency or options trading q Resource allocation for disaster relief Heavily Regulated Industries Mission-‐CriZcal Systems Life-‐CriZcal Systems Financial Systems Legacy Systems Without Automated Tests © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 5. 5 What Is Requirements Traceability? • Inventory of all requirements • …traced to features implemented in the software, • …traced to test cases that cover the functionality • …with indicator for most recent pass/fail test status Typical Artifact: Requirements Traceability Matrix (RTM) System shall… Module XYZ Test Case 123 © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 6. 6 Why Is Requirements Traceability Important? Ø Thousands of complex features Ø …developed over many years Ø …by people who are no longer on the program Ø …few automated tests Ø …a brittle code base How can we know we have tested everything? © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 7. 7 Agile Approach: Automated Unit Tests q One test generally tests one class q Written by developers q Intended to be run after every change q “Classic” unit tests: code only q xUnit extended family q Database testing q Graphical user interface (GUI) testing q Smoke testing q Performance testing © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 8. 8 Measuring Unit Test Coverage q Measures how much of the codebase is covered by at least one unit test q Typical target number: 80% © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 9. 9 LimitaZons of Unit TesZng Difficulty Creating Unit Tests q Legacy projects may not have good automated test coverage q Code may not be built for testability Only A Partial Solution q Non developers cannot understand unit tests q Coverage of code does not guarantee a feature works as needed Build The Thing Right Build The Right Thing GUI UZliZes Security / Logging Service Business Logic GUI Module Database Server Persistence Security / Logging UZliZes Service Business Logic Feature © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 10. 10 What about Test Driven Development (TDD)? Test Driven Development q A Design technique q Tests are written first in order to drive the design q Additional tests may be added later Why Does It Work? q There are generally multiple valid ways to design a piece of code q Writing a test first tends to produce more testable, flexible and modular design More Typical Alternative q Tests written during/after code q May require more refactoring to achieve desired results TDD: Good Practice; But Does Not Solve The Problem © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 11. 11 SoluZon: Behavior-‐Driven Development (BDD) q Different Process q Different Participants q Different Focus: Build The Right Thing User Story #53 Acceptance Criterion 1 Acceptance Criterion 2 Acceptance Criterion 3 Automated Acceptance Test #53.1 Given a claim has been reported… When the claim is entered… Then the system should return… Also Known as Acceptance Test Driven Development (ATDD) © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 12. 12 Feature File BDD With SpecFlow 1. Write Feature Step DefiniZons 2. Generate Template 3. C © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 13. 13 BDD With SpecFlow: Roles and Tools Developer Tester Analyst Product Owner Syntax HighlighZng Support Gherkin Plugin Syntax HighlighZng Support Gherkin Plugin PicklesDoc Document GeneraZon CodedUI GUI-‐Level AutomaZon Test AutomaZon Base Fwk (or Selenium) (or MSTest) Three Amigos Project Portal/Wiki Given… When… Then… Given… When… Then… Whiteboard © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 14. 14 BDD With SpecFlow: Process 3. Tester Completes “Happy Path” Scenario Tester Given… When… Then… 5. Developer Implements Code And Test Steps Developer SpecFlow Pair 2. Create Dra& SpecificaZon During MeeZng User Story 240 -‐ Acceptance Criteria 1) XYZ Field should be read-‐only 2) Total should match 999.99 Scenario: Ensure XYZ Field is read-‐only when… Scenario: Ensure Total is… Analyst 1. Three Amigos Discuss User Story or Bug Developer Tester Analyst PO @ignore Scenario XYZ Given… When… Then… 4. Developer Adds Dra& SpecificaZon To TFS Developer Big? Risky? Small? Fix Now Technical Product Backlog User Story 999 Cleanup XYZ…. 8. Rework, Refactoring, or Cleanup IdenZfied? 7. Review Completed Test Developer Tester Analyst 6. Tester, Analyst Add Addl Scenarios Given… When… Then… Given… When… Then… Analyst Tester © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 15. 15 GeneraZng DocumentaZon From SpecificaZons q Pickles Report Generator q Features organized In tree structure q More sophisticated commercial options available (SpecFlow+) © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 16. 16 Organizing Automated Tests: By Type q Named after class being tested q Test project structure mirrors source code product structure q Named after feature q Test project structure mirrors business domain Unit Tests Feature Tests © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 17. 17 Organizing Automated Tests: Filter By Category Category ExecuAon Speed Dependencies ExecuAon Context @CI Very Fast None: Code Only Developer, before every check-‐in @DB Slower Database structure, may use pre-‐set test data CI Build @smoke Slowest Full Running System Nightly Build @CI Scenario: Ensure abc… @DB Scenario: Ensure xyz… @smoke Scenario: Ensure 123… q Each individual test case is categorized according to its dependencies q “Run All Tests” automatically runs the correct set of tests depending on execution context © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 18. 18 Organizing Automated Tests: Test Frameworks SpecFlow tests can be wri8en at different levels – even within the same feature file Feature File Unit! ! CodedUI CodedUI Step DefiniZons – ImplementaZon AlternaZves Gherkin © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 19. 19 Organizing Automated Tests: Examples Acceptance Criteria 1. Confirm that a trade that would trigger margin call will require secondary authorization protocol 2. Confirm that a trader is not allowed to enter a quantity of more than 50,000 shares for an options trade 3. Confirm that an out of the money trade will be flagged User Story #13628 Enforce Trade Limits For Option Trades Possible Strategy Why? CodedUI Test using Moq Specific GUI is important and needs to be tested NUnit test, no DB • CalculaZon is most important asset to test, GUI may change • Legacy system is being rewripen and replacement system must support same logic NUnit Test with DB • Significant base data setup required, mocking is not pracZcal • Significant logic to be tested in the persistence layer CodedUI Test with DB Full-‐stack smoke test is desired (requires more maintenance; there should be relaZvely few of these) © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 20. 20 Extending BDD For Traceability q Epics Linked to User Stories in TFS q SpecFlow Tests Map To User Stories User Story 1098 Feature: 1098 Scenario: XYZ Scenario: ABC Epic 1608 User Story 2689 User Story 3104 User Story 7510 Epic 2387 User Story 7513 Feature: 2689 Scenario: XYZ Scenario: ABC Feature: 3104 Scenario: XYZ Scenario: ABC Feature: 7510 Scenario: XYZ Scenario: ABC Feature: 7513 Scenario: XYZ Scenario: ABC Pros 1. Simple 2. Easy to track Cons 1. User story abstraction does not really match feature 2. Requires maintenance © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 21. 21 Challenges for Maintaining Traceability Issues 1. A user story may describe only part of a true business feature 2. Later user stories may modify, contradict, or replace earlier user stories Mitigation: Let the tests help 1. A failing SpecFlow test may indicate an older specification needs maintenance 2. If the earlier SpecFlow test does not fail, consider whether it is really important to update an earlier user story with new information User Stories are designed to be ephemeral – only ac,ve within their Sprint or Release © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 22. 22 The Agile RTM q All system functions expressed as user stories q All user stories have acceptance criteria q Each criterion translated to an automated test using structured English (Gherkin) q Customized report matches epics and user stories to automated acceptance tests q Test fails unless software is implemented correctly Links to Theme Record in TFS Links to Epic Record in TFS Links to User Story in TFS Links to Feature DocumentaZon Generated via Pickles © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 23. 23 GeneraZng the Agile RTM TFS Build PowerShell Script TFS Themes, Epics, User Stories Console Runner Results of Automated Tests PicklesDoc Document GeneraZon SpecificaZons Agile RTM Project Portal / Wiki Combine & Transform XML Files Publish © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 24. 24 Building A Test AutomaZon Capability BUILD TESTING CAPABILITY q Technical Frameworks q NUnit q SpecFlow q Moq q Reporting Capability q Test Results q Code Coverage q Functional Coverage q Encouraging Adoption q Examples Everywhere q Mentoring q Pair Programming MAKE THE SYSTEM TESTABLE q Upgrade dot net framework q Drop support for old operating systems q Upgrade third party libraries q Refactor code to isolate dependencies q Simplify project structure q Different components may need different strategies Both Pieces Are Required For Success © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 25. 25 TransiZonal Strategy: StarZng with Manual Test Cases Epic Manual TestCase Manual TestCase Manual TestCaseUser Story Manual TestCase Legacy Req Legacy Req q Import legacy requirements into TFS q Convert high-level requirements into epics q Capture manual test cases in MTM q Link MTM test cases to their requirements and/or epics q Capture all new work using user stories q Link legacy system requirements to epics alongside new user stories Epic Legacy Req Legacy Req Gradually migrate legacy “Shall” requirements to user stories and acceptance criteria Plan © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 26. 26 ConverZng From Requirements to User Stories Why do this? q System has long projected lifespan q Planned re-‐plarorm, migraZon, upgrade, or significant redevelopment User Story Legacy Req 1 to 1 Legacy Req Acceptance Criterion 1 to 1 User Story Legacy Req 1 to many User Story User Story Legacy Req User Storymany to 1 Legacy Req Legacy Req © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 27. 27 Legacy System Replacement Benefits of SpecFlow for re-‐development q SpecificaZons confirm new system meets requirements q SpecificaZons represent up to half of overall effort q SpecificaZons provide documentaZon for the new system q Highlights areas where new system differs from old system q Significantly reduces risk Legacy System Replacement System Significant Reuse Gherkin SpecificaZons Step DefiniZons © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 28. 28 Summary 1. Traceability is important for mission criZcal and life criZcal so&ware 2. Automated test frameworks can provide the assurance we need ü Did we build the right thing ü Did we build it right ü Are all the required funcZons working correctly 3. SpecFlow has been proven effecZve © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394
- 29. 29 QuesZons? © Copyright Ariel Partners 2014 *[email protected] ((646) 467-‐7394 We are available for consulZng or Agile coaching. Feel free to contact us! Ariel Partners (646) 467-‐7394 [email protected] www.arielpartners.com @arielpartners Thank You!