Metrics
2 likes•1,117 views
This document discusses metrics that can be used at various phases of the product development process, including code metrics during development, test addition metrics, test execution metrics, test coverage metrics, stability trends, prediction metrics, and bug metrics. The metrics are intended to provide objective indicators of product quality, check progress, aid decisions about phase exits/entries, and trigger actions for improvement. Key metrics discussed include code complexity, test coverage, bug trends, and achieving a goal of zero open bugs for release.
Metrics
- 1. Metrics & Measurements A Product Ship Perspective Vijayan Reddy Quality Manager Adobe Systems 22/March/2007
- 2. Metrics & Measurements : Why Objective Indicators for Product Quality Check Points at various phase Watch Quality of Testing Avoiding Subjective Decisions
- 3. Metrics : Why Informative Track the Progress of Quality Improvement Aiding Decisions Are We Good to Enter / Exit a phase Triggering Action Items Bad Metrics : Fix Product Fast ! Are We Slipping? Then look where to do better.
- 4. Essential Minimal Metrics… K.I.S.S Less Overhead in Collection of Metrics Discussed Product Health Parameters Ship Readiness Metrics Not Discussed Project Management Metrics Process Efficiency Metrics Applicable for Me? No One Size Fits All Theory Apply due judgment
- 5. Dev Phase : Code Metrics Code Cyclomatic Complexity Coding Violations DPMO Automated Code Review Guidelines & Practices Naming, Comments, Indent guidelines Hard coded values, Strings Uncaught Exceptions Un-freed Memory High Logical Nesting Possible Bugs Fall through Switch Cases History Of Violations & Baselining
- 6. Dev Phase : Test Addition Metrics Traceability Matrix : Tracks Metrics Requirements Features % Tests Added as per Planned Matrix Sub Features Status of Test Addition Test Scenario % Tests Per Feature Test Case Reinforce Weightages of features on Test Data Criticality. Input for Planning Test Cycles for Place Holder for Bugs / Changelist Ids estimation of test execution tasks. % New Tests Vs Legacy Tests Test Bugs Fixed Vs Known Test Failures
- 7. Test Phase : Execution Metrics Test Passes Covered Functional Non Functional Tests Executed Pass Ratio Progress Regressions Rate MTBC / MTBF Trends Builds / Fixes Covered Platforms / Languages Covered
- 8. Test Phase : Test Coverage Metrics Code Coverage Packages / Classes Coverage Method Coverage Statement Coverage Condition / Path Coverage Coverage Driven Test Development Quality Check on Completeness of Testing
- 9. Test Phase : Stability Trends Stability T rends : Features Most Stable / Buggy Features 70 Prioritize Testing on Buggy Features 60 Feature wise Regression Incidence 50 Most Fixed / Changed Features 40 Failures 30 Most Stable / Buggy Platform 20 Prioritize Testing on Platform Builds 10 Stability 0 1xxx20 1xxx21 1xxx22 1xxx23 1xxx24 1xxx25 Builds Fe ature Wise Bugs A B C D E F G H I J
- 10. Test Phase : Prediction Metrics Bug Glide Path Prediction based on Find Rate, Fix Rate, Deferral Rate Daily Update on Actuals Predict Hitting Zero Bug Count Date Open Bug Status & Prediction (Includes actual open, new and needs Targeting) 70 60 50 40 30 20 10 0 12/15/06 12/22/06 12/29/06 01/05/07 01/12/07 01/19/07 01/26/07 02/02/07 02/09/07 02/16/07 02/23/07 03/02/07 03/09/07 03/16/07 03/23/07 03/30/07 04/06/07 Predicted Plan Combine Actual
- 11. Test Phase : Bug Metrics Daily Auto Reminders Bugs To Target Bugs To Fix Bugs To Test Deferred Bugs Bugs Incidence / Fix Trends Average Life Time of Bugs % of Re-Opened Bugs % of False Alarms
- 12. Ready to Ship? : ZBC Bug Glide Path Is ZBC achieved from Release Candidate build Any Spike in Open Bugs recently? Any Crash Landing of the curve recently? Open Bug Status & Prediction (Includes actual open, new and needs Targeting) 70 60 50 40 30 20 10 0 12/15/06 12/22/06 12/29/06 01/05/07 01/12/07 01/19/07 01/26/07 02/02/07 02/09/07 02/16/07 02/23/07 03/02/07 03/09/07 03/16/07 03/23/07 03/30/07 04/06/07 Predicted Plan Combine Actual
- 13. Ready To Ship ? : Tested Enough? Number of Test Passes Functional Stability Along the Builds Along the Configurations Along the Features All Regressions Cleared? All Failures Tracked? Code Coverage Have we Tested Enough?
- 14. Ready to Ship? : Bug Metrics Are We Finding Less Severe Bugs ? Are We Finding Less Priority Bugs? Bugs Incidence : Priority Based Bugs Incide nce : Se v e rity 25 20 18 20 16 14 15 P4 12 S4 P3 10 S3 10 P2 S2 8 P1 S1 6 5 4 2 0 0 15/02/2007 22/02/2007 01/03/2007 08/03/2007 15/03/2007 22/03/2007 15/02/2007 22/02/2007 01/03/2007 08/03/2007 15/03/2007 22/03/2007 Are We Still Fixing High Priority Bugs ? Are We Still Fixing Highly Critical Bugs? Fixing T re nd : Priority Fixing T re nd : Se v e rity Base d 20 20 18 18 16 16 14 14 12 P4 12 S4 10 P3 10 S3 P2 8 S2 8 P1 S1 6 6 4 4 2 2 0 0 15/02/2007 22/02/2007 01/03/2007 08/03/2007 15/03/2007 22/03/2007 15/02/2007 22/02/2007 01/03/2007 08/03/2007 15/03/2007 22/03/2007
- 15. Ready to Ship? : Bug Metrics Found / Fixed / Deferred : Priority Based 250 Bugs Found vs Fixed vs Deferred Are We Fixing the Right Bugs? 200 Are We Deferring the Wrong Bugs? 150 Found Fixed False Alarm Deferral Rate : Are We Deferring Our Way 100 Deferred to the Release? 50 0 P1 P2 P3 P4 Ov e rall Distribution of Found Bugs Found / Fixe d / De fe rre d : Se v e rity Base d 250 46 9% 44 200 9% 150 Found Fixed Fixed False Alarm False Alarm Deferred 100 Deferred 50 411 82% 0 S1 S2 S3 S4
- 16. Ready to Ship ? : Non Functional Metrics Performance Metrics (Product Dependent Let us assume Web Application for Eg.) Launch / Startup Time Http Response Time Concurrent Http Requests Handled Users Serviced Successful / Failed Requests Average Transaction Time Baseline / Goal Results Comparison
- 17. Ready to Ship ? : Non Functional Metrics Reliability Metrics MTBC / MTBF Scalability Memory Profiling / Foot print Footprint Size for various platforms Memory Usage : Startup, Under Load, Under No Load
- 18. Ready to Ship ? : Other Checks Security List of Risks & Mitigated Risks Memory Leaks Found / Fixed Size of Existing Leaks Beta / Customer Feedback Issues Reported / Fixed Open Issues Incidences Trend Since Beta
- 19. Ready to Ship ? Guess SO !
- 20. Questions
- 21. Thanks! References Wikipedia : Hammurapi Wikipedia : Cobertura, Emma, JCoverage