Agile Metrics and Business Value
– Harish Vasista, PMP, PgMP
It is a fact that more and more projects are being done according to the Agile model. Those who want to switch over to Agile, commonly ask about its benefits, as compared to traditional models. Management will want to know the business impact of Agile. These valid questions need to be answered effectively to convince that Agile is the way forward. Metrics other than the burn-down chart and the velocity of the team need to be identified. Care must be taken to measure the outcome and not the process. Albert Einstein rightly said, “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”
So, let’s look at some of the common metrics that can be collected during an Agile project.
1. Quality/Defect Containment
A measure of the quality of the product/release. The entity measured is the number of defects reported by the customer. This gives a clear indication of how good the development process is in catching the defects earlier in the cycle.
How to Measure : Keep track and count the number of defects reported by the customer per release. This can be a negative value depending on the number of defects reported by the customer. Care must be taken to include only the defects that are relevant to the product. There will be cases where customer-reported defects may not be directly relevant to the product developed (examples: fiber cuts, power outages, problems due to other elements in the network, etc.).
Defect containment (DC) = [1– (Dc/Di)]×100%
Dc = Number of priority 1 (P1) and P2) defects detected by customers for a release
Di = Total number of defects detected internally
2. Feature Waste
A measure of the amount of effort spent by the team on features that are “wasted” (features that the customers are not interested in). This is an important metric, as elimination of waste is one of the big ideas in Agile.
How to Measure : This is relatively easy to track, based on inputs from the Product Owner. It is measured as the number of features that are not used by the customer and expressed as a percentage of the totality of the feature set available in the product.
Feature waste (FW) = [1 – Fu/Fa] ×100%
Fu = Number of features or effort spent on features used by customers in a release
Fa = Total number of features or effort spent on features delivered in the release
3. Flexibility Index
A measure of the agility of a team. The entity measured is the amount of undone work left at the end of the last sprint. The rationale of measuring this is to determine how quickly the team can react to market changes. If the amount of undone work is high, it means that the team is not Agile enough. The lower the amount of undone work, the higher the flexibility index.
How to Measure : The actual amount of time in weeks required to complete the undone work. Measure the amount of time from the end of the last sprint. It is expressed as the ratio of the undone time to the total duration of all sprints (number of sprints sprint duration).
Flexibility index (FI) = [1 – UT/(Sn Sd)]×100%
UT = Undone time (in weeks)
Sn = Total number of sprints in the release
Sd = Sprint duration (in weeks)
4. Customer Satisfaction Index
The traditional old-school metric is what we ultimately strive for. It is an indirect measure of how the changes in the development process have impacted the customer. It should be noted that customer satisfaction is dependent on a lot of other factors, such as service quality, on-time delivery, response to issues, etc. But it should continue to be measured to ensure that Agile does not adversely impact customer satisfaction.
How to Measure : Measured on a scale of 1-10, based on end-customer feedback. Customer satisfaction surveys can be conducted to arrive at this metric.
5. Efficiency Index (EI)
A measure of the value-add (VA) time in a project. In traditional models of development, the efficiency index is 2-5% due to the immense number of pre and post-development activities. It should be significantly higher in the Agile model, as value is continuously added to the product.
How to measure: Keep track of the amount of time that value is added to the project. These activities include design, development and end-customer documentation. The EI is the ratio of the VA time to the total time for the project.
Efficiency index (EI) = (Vt /Tt )×100%
Vt = Value-add time in weeks
Tt = Total time/duration of the project in weeks
Note: There should be a clear definition of what activities contribute to value addition. For example reviews, testing, and fixing bugs found in testing should not be considered as valueadding activities.
6. Early Defect Detection Index (EI)
A measure of how effective the iterative development process being followed. If the process is really effective, a higher number of defects tends to be detected earlier in the cycle.
How to measure: Keep track of all the P1 and P2 defects detected during the sprints, as well as the P1 and P2 defects detected during the undone phase of the release.
Early defect detection index (EDDI) = (Ds/Du)×100%
Ds = Total number of P1 and P2 defects detected during regular sprints
Du = Total number of P1 and P2 defects detected during undone phase
There are other business-value metrics, such as EBV (earned business value) and RTF (running tested features), which try to quantify the business value delivered in each sprint. These metrics may not be applicable as long as we have a significant amount of undone work planned at the end of the last sprint.
In summary, the six metrics explained above, in addition to the velocity and the burn-down chart, provide views of both the status at a given time and also the trends being established – without their measurement adding significant overhead effort. It is not necessary to measure all six. Scrum teams can select the metrics that make sense to them and are applicable for their projects.