Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Innovating with Emergence - EuroSTAR 2012
3 likes1,162 views
This document discusses value-inspired testing and innovating testing approaches. It proposes renovating risk-based testing by collating current risk-based testing variants, using a context-driven mix of risk principles, grading test coverage rather than truncating tests, and balancing risk against benefits to provide net value. It also suggests innovating testing by considering evolution in nature as a value flow, appreciating the concept of memes and evolving "memeplexes" in testing, and finding an emergent path between too much chaos and too much order. The document argues that by taking a holistic and evolving approach, testing will continue to innovate rather than become obsolete.
Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Innovating with Emergence - EuroSTAR 2012
- 1. Value-Inspired Testing: Renovating Risk-Based Testing, and Innovating with Emergence Neil Thompson, Thompson information Systems Consulting Ltd www.eurostarconferences.com @esconfs #esconfs
- 2. Value-Inspired Testing v1.1 Renovating Risk-Based Testing, and Innovating with Emergence 2 Neil Thompson © [email protected] @neilttweet neiltskype
- 3. Deming: survival is “not compulsory” • Tim Rosenblatt (Cloudspace blog 22 Jun 2011) “Testing Is Dead – A Continuous Integration Story For Business People” • James Whittaker (STARWest 05 Oct 2011) “All That Testing Is Getting In The Way Of Quality” • Alberto Savoia (Google Test Automation Conference 26 Oct 2011) “Test Is Dead” • (There *may* be others?) Are reports of testing’s death “greatly exaggerated”? 3
- 4. But those definitions of testing seem too narrow – my Agenda instead... • To renovate the use of Risk in testing: – collate current variants, eg “Risk-Based, Risk-Driven” – use context-driven mix of principles – grade testing from high to low (not truncate) – balance risk against benefits, giving net Value – use risk throughout testing “process” – integrate risk into SDLC using Value Flow ScoreCards • To innovate in testing: – consider evolution in Nature – also a value flow? – appreciate concept of Memes; evolving “memeplexes” in testing – emergent path between “too much chaos” & “too much order” – creativity: where good ideas come from (Johnson) 4
- 5. So, when holistic & evolving, testing will not die? (based on http://www.needham.eu/wp-content/uploads/2011/01/ascent-of-man1.jpg) 5
- 6. Start renovation of “Risk” by collating current variants 6 1976 IMPLICIT RISK PRINCIPLES “TESTING IS RISK-BASED” HOW TO DO IT Risk as prioritisation of features etc RISK-BASED TEST MANAGEMENT RISK-BASED TEST DESIGN Risks as entities to test, driving techniques RISK, SCHMISK! 2002 ! 1972-3 1970s - 1984 1990 1979 1984-1988
- 7. Use a context-driven mix of available principles 7 RISK-BASED TEST MANAGEMENT Quality criteria Quality criteria Business risks Risk factors to choose, eg: • usage • newness • complexity Technical risks RISK-BASED TEST DESIGN Project environment Prioritisation Perceived quality Project environment Risk workshops: • why, whether, who, where? • when, what risks, how handle? After: Heuristic Test Strategy Model v4.8, James Bach Test techniques Product elements Product elements What to prioritise & focus on: • test items? • features? • data items? • test conditions?
- 8. Prioritisation: better than truncating “low-risk” tests, *grade* coverage 8 Test Coverage & Effort Even distribution Riskiness Random / spurious priorities Risk-truncated X X After: Chris Comey, Testing Solutions Group Risk-graded X • Does this make sense? • No! • Even less sense! • Better, but dangerous to omit some areas completely? • This is the most responsible way
- 9. Consider not only risks – balance against benefits to give net value Priorities Tests graded by... 9 Business... + FEATURES etc ... .. .... . ..... ... .. ... . Open Closed Product Risks Open Closed Closed Open Open Benefit Benefit Objective Objective Objective Objective Benefit Objective Closed Project objectives, hence business benefits, available for release now After: Paul Gerrard & Neil Thompson, book Risk-Based E-Business Testing Objective Objective Objective Objective Objective Objective Objective Project... .............Value
- 10. Apply risk principles throughout software lifecycle validation testing verification testing 10 TEST MODEL programming with risk of bugs simplification Requirements Functional Specification Technical Design Module Spec Acceptance Test Analysis & Design System Test Analysis & Design Integration Test Analysis & Design Component Test Analysis & Design DEVELOPMENT MODEL REAL WORLD refinement with risk of distortion AT Execution ST Execution IT Execution CT Execution SOFTWARE DEV MODEL (expected) TEST MODEL (ver’d / val’d) SOFTWARE (observed) REAL WORLD (desired) after SOFTWARE TESTING: A CRAFTSMAN’S APPROACH Paul Jorgensen So: • remember overlapping models • we need both verification & validation • this is not “the” V-model!
- 11. Bear in mind causes and effects of risks 11 simplification Requirements Functional Specification Technical Design Module Spec programming with risk of bugs TEST MODEL DEVELOPMENT MODEL REAL WORLD refinement with risk of distortion Validation SOFTWARE Mistake: a human action that produces an incorrect result (eg in spec-writing, program-coding) Defect: incorrect information in specifications Fault: an incorrect step, process or data definition in a computer program (ie executable software) On TEST “process” Anomaly: an unexpected result during testing Knock-on Effects Failure: an incorrect result Error: amount by which result is incorrect Probability of making mistakes, of defects causing faults, faults causing failures, etc Consequence of risk if it happens............................................................................... On REAL WORLD after go-live Static Verification
- 12. Risk principles apply throughout testing “process” Static validation Static verification May be all or partially exploratory................. Detect further bugs; Adjust test coverage On DEV & TEST “processes” On REAL WORLD after go-live 12 other oracles Specification TEST MODEL DEVELOPMENT MODEL Prevention Mistake Defect Knock-on Effects Fault Anomaly Failure Error Detect omissions, distortions, rogue additions... Use “Peopleware” principles Test analysis Test design Test exec’n Bug mgmt Prioritise by both urgency............. & importance................................ Fix, test fixes, regression-test Write / model better requirements
- 13. A framework for managing value through the lifecycle: “Value Flow ScoreCard” 13 Financial Supplier Improv’t Customer Process Product WHO... Supplier Customer Financial Infra-structure Process Product WHY WHAT, WHEN, WHERE HOW • In action, the ScoreCard is a 7x4 table: – uses include setting / balancing test policy, strategy, coverage, troubleshooting & improvement – can start with repositionable paper notes, or use spreadsheet – NB the measures & targets need not be quantitative, may be qualitative eg rubrics Improve-ment Infrast • “The seven watchwords of highly effective software people!”
- 14. Risk can be integrated into the scorecard 14 Objectives Threats to success Measures Targets Initiatives WHY we do things HOW they may fail WHAT (will constitute success, WHEN & WHERE) HOW to do things well SEVEN VIEWPOINTS of what stakeholders want Supplier Customer Financial Improvement Infrastructure Process Product Risk Risk Risk Risk Risk Risk • Now it’s a 7x5 table
- 15. Types of risk 15 Project risk Process risk Product risk Eg: • supplier may deliver late • key staff may leave may cause Eg: • configuration management may install wrong version of product Eg: • specifications may contain defects • software may contain faults may cause may cause may cause
- 16. So: we’ve renovated “risk-based testing” into a whole-lifecycle structure 16 Objectives Threats to success Measures Targets Initiatives WHY we do things WHAT (will constitute success, WHEN & WHERE) HOW to do things well SEVEN VIEWPOINTS of what stakeholders want Supplier Customer Financial Project risk Process risk Product risk (Process risks) Project risk Project risk Infrastructure Process Product Improvement
- 17. Now to move on to innovation 17 • The double feedback loop of the ScoreCard: – not only is our – but also: how we are planning to improve for next & future projects Objectives Threats to success Measures Targets Initiatives Process Product Improvement Process risks Supplier Customer Financial Infrastructure scorecard, and its cascading, converging on desired targets for current project...
- 18. How does Nature innovate? Lamarck: Acquired characteristics, Usage, Inheritance Darwin: Mutation, Fitness, Reproduction (various authors) Emergence... Images from wikipedia 18
- 19. A scientific view of emergence 19 Physics (quantum end) Chemistry: Inorganic Chemistry: Organic Biology Sources: Daniel Dennett “Darwin’s Dangerous Idea” Physics (gravity end) Social sciences “cosmic Ouroboros” (Sheldon Glashow, Primack & Abrams, Rees etc) Image from http://www.aaas.org/spp/dser/03_Areas/cosmos/perspectives/Essay_Primack_SNAKE.GIF (Ouroboros: Greek Οὐροβόρος or οὐρηβόρος, from οὐροβόρος ὄφις "tail-devouring snake”)
- 20. Is like value flow? (and it looks better this way up!) 20 • Each level of progress generates possibilities, which are tested • Then, each level is a platform which, when established, is easily built upon by “cranes” (without having to worry about the details below) • After the science levels... • humans made tools, talked and co-operated • printing gave us another level • now, software is following exponential growth • So, software testing should surf the wave of evolution (not flounder in the shallows behind it) • Kurzweil epochs 6: Intelligence into matter/energy patterns? 5: Bio methods integrated into technology? “SINGULARITY” 4: Technology 3: Brains 2: Biology 1: Chemistry & Physics +0: Maths?! The Singularity is Near, 2005
- 21. The Darwinian view of evolution – but does this explain all emergence? Image from www.qwickstep.com 21
- 22. Biological evolution as sophistication rising with diversity 22 Sophistication Time Diversity
- 23. But evolution is not smooth? Spread into new niche, eg Mammals Mass extinction, eg Dinosaurs 23 Sophistication “Gradual” Diversity Darwinsim Sophistication (equilibrium) Punctuated Diversity equilibria “Punctuated equilibra” idea originated by Niles Eldredge & Stephen Jay Gould Images from www.wikipedia.org “Explosion” in species, eg Cambrian (equilibrium) (equilibrium) Sophistication Number of species Diversity
- 24. So... evolution of sciences overall? 24 Biology Organic Inorganic Chemistry Physics Social sciences • Arguably other sciences have not evolved smoothly either • Sudden advances, akin to punctuated equilibria in biological evolution Per Bak, “How Nature works” 1996 (image Tracey Saxby, Integration and Application Network, University of Maryland Center for Environmental Science ian.umces.edu/imagelibrary/) Sophistication Diversity
- 25. OK, what’s all this got to do with software testing? • Social sciences evolution Tipping Points (Malcolm Gladwell) Sophistication Tools Language Books Computers • We have an important and difficult job to do here! Diversity Social sciences 25
- 26. Testing needs to evolve / emerge / innovate to keep up with complexity Sophistication 2GL 1GL Object Orientation Internet, Mobile devices Artificial Intelligence?! 4GL 3GL • For example, are we ready to test AI?? Diversity Computers 26
- 27. How has testing evolved so far? “no schools, but...” 27 PERIOD 1957 1976 EXEMPLAR OBJECTIVES SCOPE “SCHOOL”? Pre- 1983 1984 2000 2011 Weinberg (1961 & 71) Test + Debug Programs Hetzel Show meets (1972) requirements Myers Find bugs (1976 & 79) Programs Programs, System, Accept’ce ? Kaner et al (1988 & 99) Experiment & Evolve? Neo- Holistic? Measure quality ? Beizer (1984) + Integr-ation Find bugs, show meets requirements, +prevent bugs Find bugs, in service of improving quality, for customer needs Standard (Control) Analytic Quality Context Driven ? Agile Factory (Test-D riven) DEBUGGING (Psychology) DEMONSTRATION (Method) DESTRUCTION (Art) EVALUATION (Engineering?) PREVENTION (Craft?) AUTOMATION? (Technology?) HUMANISATION? (Social Science?) UNIFICATION?? Science? Overall periods developed after Gelperin & Hetzel, “The Growth of Software Testing”, 1988 CACM 31 (6) as quoted on Wikipedia
- 28. Another way of thinking about evolution: genes... Sophist-ication Diversity Replication & Selection Mutation Image from www.qwickstep.com Image from .schools.wikipedia.org 28
- 29. ...and for humans, “memes”, as an extension of the genes concept Symbols Ideas Beliefs Practices 29 “Other imitable phenomena” Writing Speech Rituals Sophistication Mental, social & cultural evolution Biological evolution Theme developed from Daniel Dennett “Darwin’s Dangerous Idea” Platforms Cranes (Lamarckian??) Replication & Selection Mutation Gestures Image from .www.salon.com Taxonomy from www.wikipedia.org Diversity
- 30. Considering memes in testing: here is an example “memeplex” Assess where errors originally made 30 Effectiveness Always-consider Efficiency Risk management Quality management Insurance Assurance Give confidence (AT) Define & detect errors (UT,IT,ST) Be pragmatic over quality targets Use handover & acceptance criteria V-model: what testing against W-model: quality management Risks: list & evaluate Prioritise tests based on risks Tailor risks & priorities etc to factors Use independent system & acceptance testers Refine test specifications progressively: Plan based on priorities & constraints Design flexible tests to fit Allow appropriate script format(s) Use synthetic + lifelike data Use appropriate skills mix Allow & assess for coverage changes Document execution & management procedures Distinguish problems from change requests Prioritise urgency & importance Distinguish retesting from regression testing Define & measure test coverage Measure progress & problem significance Quantify residual risks & confidence Decide process targets & improve over time Define & use metrics Define & agree roles & responsibilities Use appropriate techniques & patterns Plan early, then rehearse-run, acceptance tests Use appropriate tools Optimise efficiency Source: Neil Thompson STAREast 2003 (not “best practices” but reference points for variation?)
- 31. Another example memeplex for testing 31 • (Grouped here by chapter for illustration, and coloured by theme) • 293 individual “lessons” selectable by testers according to context Management Managing the testing group Managing the testing project Planning the testing strategy Interacting with programmers Source: Neil Thompson BCS SIGiST 2002 review of Lessons Learned in Software Testing (Kaner, Bach & Pettichord) Your career in software testing The role of the tester Thinking like a tester Testing techniques Bug advocacy Automating testing Documenting testing Thinking
- 32. So, do we have punctuated equilibria in the evolution of testing? • Where were the UNIFICATION?? Science? Platforms? • What were the CRANES? • Tipping points? Mass-market software HUMANISATION? eg Context-Driven school Social Science? Open-source tools AUTOMATION? eg test-driven development Belief in cost-of-failure curves PREVENTION eg reviews, root cause analysis Technology? Craft? Publication of ANSI/IEEE standards EVALUATION eg metrics initiatives Establishment of textbooks Acknowledg’t of testing as distinct discipline Sources: Gelperin & Hetzel 1988 Engineering? DESTRUCTION eg test techniques Art DEMONSTRATION eg V-model Method Software analysis DEBUGGING etc?? Psychology • But... is there something wrong with this picture?... Sophistication Diversity Software testing 32
- 33. One of the existing views of innovations in software testing • Concepts: – hierarchy – products / processes • Factors: – invention / application – individuals / organisations – bottom-up / top-down – synthesis of precursors – adjacent possibilities – role of testing! • Aids: – population size – diversity / interdiscipline – free time / free to fail – psychology & serendipity – recording media 33 Testing & Quality Testing (20th C) After: Lines of innovation in software testing, Stuart Reid 2010/2011, testing-solutions.com Testing innovations in specific subjects
- 34. Arguably, emergence is more than just Lamarckian / Darwinian 34 • Emergences at coarser scales not explained by “reductionism” to finer scales • For best innovation & progress, need neither too much order nor too much chaos • Examples: galaxy development, phase transitions, Gaia, autocatalysis, aminoacids→proteins, political swings, AI & IA? Physics Social sciences Chemistry Biology Extrapolation from various sources, esp. Stuart Kauffmann, “The Origins of Order”, “Investigations” • Might also apply to testing?? Sophistication Diversity Time
- 35. History of testing is intertwined in “ecosystems” with technology, software lifecycles, etc mature Agile? 35 Testing & Quality Technology Development Method Psychology Engineering Art Craft Social science Structured methodologies CASE tools immature Agile Diversity Sophistication
- 36. And within testing, different contexts have so far evolved in separate streams? 36 Testing & Quality: TRADITIONAL “SCHOOLS” Engineering Craft CONTEXT-DRIVEN Psychology Method Art Technology Social science X X X X • Recent changes regarding “school” & “approach” • Limited dialogue, mutual mistrust, “language” differences Diversity Sophistication
- 37. An “emergent” view of innovation 37 • Eight related ideas from history of human innovation 7. Platforms Web 1. Adjacent possible City Reef 2. Liquid networks “0” 6. Exaptation 5. Error 4. Serendipity 3. Slow hunch Johnson’s ideas overlaid here on Neil Thompson’s graphic
- 38. Emergent view: (a) innovation framework • Once a new level is established, can build on it, almost without thinking 38 7. Platforms Web 1. Adjacent possible City Reef 2. Liquid networks “0” • Coral reefs surprisingly diverse habitat, because crowded, wave-washed boundary zone • Cities concentrate minority interests where they can communicate • Tech innovations used to take 10 years; on www 1 is enough “Patterns of innovation are fractal” • Things happen wherever they can happen • Ideas flowing without friction
- 39. Emergent view: (b) innovation “techniques” 39 7. Platforms Web 1. Adjacent possible City Reef 2. Liquid networks “0” 6. Exaptation • Modifications can be hi-jacked for unexpected things (and beneficially) 5. Error • Noise can make us focus more • OK to fail, but try to fail fast 4. Serendipity • You may find something different, but it’s important to be seeking something 3. Slow hunch • Many innovations are not eureka moments, they take time to evolve & establish tattoos99.com
- 40. A brief history of human innovation 1800-current Market • Rise of market communities, eg: – radio (Marconi, Tesla, Braun, Hertz etc) • Rise of amateur communities, eg: – Milky Way (Al- Biruni, Galileo, Herschel & his sister) Individual(s) Communities 40 1600-1800 1400-1600 Amateur • Most discoveries “amateur individuals”, eg: – supernovae (Brahe) Source: Steven Johnson, “Where good ideas come from: the natural history of innovation”
- 41. So, what could software testing learn from the history of innovation? HUMAN HISTORY SOFTWARE TESTING Reef, City, Web • Even if introvert, use LinkedIn, Twitter etc Adjacent possible • Try modifying / combining / hybridising Slow hunch, Exaptation techniques. They’re not set in stone (eg 2-D classification trees) • Keep a notebook. You never know what may come in handy eventually (see also Jerry Weinberg’s Fieldstone method) Serendipity • If a trail goes cold, turn your nostrils in some other Platforms • Seek new uses of previous achievements, eg test communities (market & amateur) direction automation in new ways (high-volume random) • Even competitors in this market seem to collaborate and mutually-respect. Keep it up! • Attend conferences etc 41
- 42. An additional thought Renovated risk, & Science, as UNIFICATION? 42 • Testing contexts will of course continue to differ, but... • More mutual dialogue may increase innovation, both sides Traditional, risk-averse sectors Market-chasing, product-oriented, risk-tolerant / risk-embracing sectors • ...if we can all share understanding across varied contexts Diversity Sophistication
- 43. A brief history of testing innovation? 2012 onwards? Market • Communities interacting more? Quality Analytic • Communities in relative isolation? Individual(s) Communities 43 2000-2012? 1950s-1999? Amateur • Guru individuals? Context Driven Agile (Test-D riven) Factory
- 44. Key references & acknowledgements (NB this is not a full bibliography) • Use of Risk in testing (yes, other sources are available!): – Kaner, Bach & Pettichord: Lessons Learned in Software Testing – Craig & Jaskiel: Systematic Software Testing – Gerrard & Thompson: Risk-Based E-Business Testing • Principles contributing to Value Flow ScoreCard: – Kaplan & Norton: The Balanced Scorecard – Translating Strategy into Action – Isabel Evans, Mike Smith, Software Testing Retreat • History & innovations in testing: – Gelperin & Hetzel: The Growth of Software Testing – Meerts: testingreferences.com incl. timeline – Stuart Reid: Lines of Innovation in Software Testing • Emergence: – Dennett: Darwin’s Dangerous Idea – Eldredge & Gould: Punctuated Equilibria... (in Models in Palaeobiology) – Kauffman: The Origins of Order, Investigations etc – Johnson: Where Good Ideas Come From – (+Kurzweil: The Singularity is Near?!) 44
- 45. Takeaway ideas • All testing is risk-based/value-inspired: whether or not you recognise it yet (so, make a virtue of it) • Embrace diversity; discuss! don’t dismiss, disrespect or just “agree to differ” • Mix with lots of non-testers • Seek out analogies & metaphors • Depending on your personality: – Read lots of books (eg “things to read together” = adjacent possible) – Do lots of thinking – deliberate & unintended – Participate in blogs, discussion groups • Remember: change is accelerating, and innovation is fractal! 45