Agile Metrics

Editor's Notes

• #7: When you’re on a long project – 6 months, a year or longer, we need someway to gauge these things.Developing software is a complex system that is mostly intangible. So we use these measurements as a window into that world. What’s going on here? When will we be done? What’s our quality like? Etc.It’s human nature to explain things we can’t see.
• #8: What do you think about this metric? Actually it’s a really bad one – there’s correlation/causation errors going on, and overall “project success” is way too complicated a system to judge based on one metric.Chaos Report from 1995 to 2010: project success rate goes from 16% to 30%
• #9: In waterfall we need gauges and indicators and ways to predict the future, because it’s scary to be on a project with a really long time horizon.“We all have a need to understand, we all get anxious when we don’t, we all look for ways to explain things that aren’t easy to explain. That’s what these metrics do. And, if you’re the owner of the project, your butt is on the line, so all the more to be anxious about, all the more to try to make the intangible tangible (which is what I think of software development – until you see a product, it is an intangible and intangibles are scary).” –LL
• #10: So before we talk about how Agile responds to that, let’s look a bit at how we operate as humans, and how metrics can effect our behavior.This section has one slide of theory and one real life example.
• #11: “There are so many possible measures in a software process that a random selection of metrics will not likely turn up something of value” – Watts Humphrey Metrics used in isolation probably don’t measure what you think they do.-System is more complex than this. We’re probably not ever going to be able to measure enough to give us a simple indicator of the system. - Isolated metrics entice people to draw system wide conclusions.-> Primary/Secondary MetricBeware long hanging fruit. Also, old literature praises low hanging fruit!-> Just because we can measure something easily doesn’t actually mean it’s meaningful.
• #12: Ask: Does everyone agree this is a easy to gather metric? What is this metric really telling us? Stakeholders: “How come we have less tests than a few sprints ago? That can’t be right. We must not be testing enough.” Stakeholders: “On my last project we had thousands of tests, why are there only a couple hundred? That can’t be right, we must not be testing enough, I bet this thing is littered with bugs.”This is an example of things that are easy to measure, and things measured in isolation. The system – the software development machine – is far too complex to be making broad quality statements based on such isolated measurements. But we’re so used to doing that. So you can start to see that some traditional metrics might not really fit the bill. Let’s go on
• #13: Explain Hawthorne Experiment. Select group of workers old they were being studied, and their productivity changed. All the researchers did was minutely change the lighting levels.For example, measuring test pass/fail status always causes pass percentage to rise. But it is an artificial rise, due to people not wanting to fail tests or splitting up tests into smaller and smaller units to drive the percentage calculation up (which is just creating waste). Also called demand characteristics: refers to an experimental artifact where participants form an interpretation of the experiment's purpose and unconsciously change their behavior to fit that interpretation
• #14: I’ve changed the exact wording here to protect the innocent. But here’s a good real life example. Read these two statements and think about what may have changed in the time between these two statement.
• #15: Robert Austin. Measuring and Managing Performance in Organization. Nucor Steel. Based plant managers salaries on productivity – of ALL plants, not just theirs.The obvious example here is defect counts.Edward Demming, the noted quality expert, insisted that most quality defects are not caused by individuals, but by management systems that make error-free performance all but impossible.Eh… Attributing defects to individuals does little to address the systemic causes of defects, and placing blame on individuals when the problem is systematic perpetuates the problem. By aggregating defect counts into an informational measurement, and hiding individual performance measurements, it becomes easier to address the root causes of defects. If an entire development team, testers and developers alike, feel responsible for the defects, then testers will tend to become involved earlier and provide more timely and useful feedback to developers. Defects caused by code integration will become everyone’s problem, not just the unlucky person who wrote the last bit of code. 
• #16: So what do you think happened here? What was the result?Perhaps their intense focus on defects per person, lead to no focus on the customer… perhaps not, it’s too complex to really tell, but the point is that they are probably measuring too low. Are defects-per-person-hour really important to your goal? Probably not. Measure one level up, maybe defects reported by customers…
• #18: So we’re at the point where we know that waterfall feels risky, and we know there are some behavioral aspects to metrics that we need to consider.
• #19: Agile takes all the worry and all that risk and packages it up into cute little time boxes. Agile inherently limits risk. Even if one of these boxes explode, the project isn’t a failure. And every few weeks we produce a valuable increment of product, we have the chance to inspect it and adapt our approach, reprioritize, replan etc. Managers no longer need to be worried about and have this anxiety over predicting project performance over months and months. We have real tangible results every few weeks. We can inspect it and determine the ACTUAL characteristics of the product that we used to use metrics to try to get at. Agile Projects inherently limit riskTime Boxes, WIP, DoD, AC, fast feedback(lead in) So that’s nice, but how do you define quality on this increment and on the product as a whole?
• #20: Two ways. In on any single increment we use the above mindset. These are not strict equations, I’m not doing any math here, it’s just a way to think about quality in the agile world. DoD: Shared definition among the team of what “done” means. Typically you see things like coding standards, unit test coverage, tests pass, deployable, reviewed, etc. Every piece of work must adhere to the DoD.AC: Product Owners business-language criteria for how a specific piece of work must function. Sometime written in the GIVEN-WHEN-THEN format, a practice associated with ATDD. So as we string increments of working software together, how do we get at the quality of the product? We use the mindset at the bottom for this.On the product level, it’s no longer so much about defining quality in a quantitative sense as it is about having a development process that can easily react to change. React to negative customer feedback as well as suggestions for new features and what’s most important to the customer at the moment.Stakeholders that don't show up at the Sprint Review will still be nervous, and rightly so.  The corollary is: every time a manager/stakeholder/etc. asks for a report, instead of giving it to them stress the importance of showing up at the Sprint Review.
• #21: You have clear development principles that help limit risk (DoD) (verification) and clear business objectives that help limit risk (Acceptance Criteria ) (validation). This ensures some base level of quality in your product, and then through frequent stakeholder and customer feedback, we ensure ongoing quality and value of product. Our chief metric in scrum is working software. That said, what other metrics do we need? Right?
• #22: Agile does indeed negate the need for many traditional metrics. It certainly helps make the complex rather intangible process of developing software a bit more tangible – one increment at a time.I do suggest starting here. It’s less dangerous than starting with metrics carried over form waterfall. Rip it all down and build it back up.But there are some useful metrics that we could use, so to set that context…
• #23: In his 14 Points, Deming said “Eliminate management by numbers and numerical goals. Instead substitute with leadership.”  The more we rely on metrics to tell us what happened, the more we distance ourselves from the actual work being done. We realize that measuring a system as complex as the software development machine, doesn’t really provide understanding, just data. Sometimes bad data, sometimes good data. And we realize that the obvious answer isn’t always right – like blaming bad developers for buggy products – “it must be the developers” – we respect that there is likely more going on in the system than any one root cause of anything. Further, if we use metrics the wrong way, we build games and systems that reward paying attention to the metric and not the success of the company.Overall we believe that being agile is important to the goal – our goal being making really good software products that have high value and delight customers. So we will use metrics that help us be agile. That encourage us to embrace lean and XP and good development practices.
• #25: Trends over static numbers: tear the labels off the y axisIs this setting up stakeholders to draw a system conclusion based on an isolated metric?No single prescription – figure out what makes sense for you. Take these considerations into account. We’ll go over a bunch of possible metrics next, but I’m not advocating a simple recipe for anyone. I’m certainly not saying you have to use all of these.
• #26: Our chief metric is working software. Did we get to the end of the sprint and have potentially shippable product? How do you measure this? A simple thumbs up or thumbs down. Get everyone in a room and do it. Not good enough? Then document it. We keep a running go/nogo document.Why not just do this in waterfall? Get everyone in a room at after a year long project and give it the thumbs up? Well in some sense you do – we often ignore all those other metrics we’ve spent so long gathering. We rationalize sev 1’s down to 2’s, etc. In agile you can do this more safely because YOU HAVE CONTEXT. You have really good context and memory within a timebox. The risk is limited.
• #27: Indicates team progress. A way to visualize what’s done and what’s WIP and what’s left to do. Tool to use to see when we’ll be DONE with a particular chunk of value.Don’t like hours? Don’t want a graph? Fine: use a task board, count tasks, stories-to-done, whatever. It’s just a tool so that you as a team know how work is progressing, and can visualize that and discuss it as a team.If it’s not given to management, there is little risk of negative hawthorne effect or gaming.
• #28: Not individualsNo comparing across teamsNot really for management, certainly not for incentives (risk of gaming)
• #29: Helps the business know when a larger chuck of functionality might be DONE. Not really part of scrum but also something you usually can’t get away without doing. At least this method of planning is based on empirical evidence of past sprints velocity and what’s actually on the backlog now, and also look at the cone of uncertainty there – we’re not promising a date, we’re just giving a forecast as accurately as we can while still being able to sleep at night.Increments are great, and this tells us when enough increments put together will satisfy some large business objective.
• #33: Unit Testing is a great development practice. If we measure it, we just might encourage that behavior. Pick a Target, Should never go down
• #34: Don’t discourage check-ins by making this visible at too high a level. Individuals need fast feedback, and sometimes teams can use this in-good-spirits, but it can start to deter checkins
• #35: Beware the “math” on this one – as software matures and ceases to change, this percentage approaches 0. But 0 in any one sprint indicates a problem. Rapid fluxuation might indicate some churn our lack of vision around testing (or churn in the software)
• #36: Etsy – optimize everything for employee happinesshttp://happiily.comEncourages self-awarenessLeading indicatorConfidence? When you check in or move something to doneScale is 1-5. We measure this continuously through a live Google Spreadsheet. People update it approximately once per month.Here are the columns:NameHow happy are you with Crisp? (scale 1-5)Last update of this row (timestamp)What feels best right now?What feels worst right now?What would increase your happiness index?Other comments
• #37: Running Tested Features – XP practicePositive Hawthorne Effect: We want to deliver more value (but beware gaming – you still have to be DONE)Measures up: delivered value for the product (not single team or individual)Little’s Law: queue size ~ queue time
• #38: Here’s one for your metric walls – what are the top three most common customer complaints. Or the three hottest issues right now? Post these on a wall where everyone can see them.
• #39: Size of bubbles are TCO (total cost of ownershipHopefully in a single project you are up in the magic quadrantAcross a program/product there might be some things in other places – “have to do’s” compliance and legal stuff…
• #40: For a single feature, you can also drill down one level and look at the number of times per day/week/month a user uses it, and the amount of time spent using the feature.Why measure this? Are we building the right features? Is a bug in feature “C” more critical than a bug in feature “I”? Feature “K” may have more maintenance costs than value – consider dropping it.
• #41: Where is there waste in the system?What’s the best time for a nominal task?