When people talk about metrics, they often seem to lump them all together into a single sort of thing. This lack of distinction often seems to generate needless confusion. In my experience, it is helpful to have an awareness of the following broad classes of metrics before diving into a metrics discussion.
Progress Metrics: These are numbers that help demonstrate an individual or team’s progression towards completion of a task or project. Burn down charts, as well as Earned Value Measures, are examples of this sort of thing. Progress metrics help a team and its leadership understand whether they are on track to complete a task per their plan. If your actuals show a significant variance from your plan, then that is typically a trigger for some sort of corrective action or a replan.
Note that these sorts of metrics often count completions of tasks and/or deliverables. In this sense, they are telling us about the means to an end, and not the ends themselves.
Health Metrics: These are numbers that tell us how well or poorly we are doing in some area. For example, a customer satisfaction survey may tell us that 70% of our customers are happy with our work. Whether that is a good thing or a bad thing would depend on whether it represents an upward trend or a downward trend, or whether it is higher or lower than scores achieved by our peers, or how it compares to some organizational goal that has been established.
Note that these sorts of metrics are focused more on results than on progress towards a goal. In this sense, they tell us how well we are achieving our ends.
A common complaint about health metrics is that they don’t often seem to spur useful action. If we are doing well, then management throws us a party; if we are doing poorly, then we receive a stern memo. However, these sorts of metrics offer no clue as to what we did right or wrong in order to earn the praise or blame bestowed upon us, so there is no guarantee that future performance will improve.
The commonly heard saying that “The beatings will continue until morale improves” seems to reflect a typical frustration with health metrics and they ways they are employed. The underlying truth being satirized here is not that managers abuse employees, but that low scores on any type of metric tend to result in grumpiness and unpleasantness from management, and that these reactions are usually counterproductive.
Health metrics are often tracked on scorecards that are closely watched by executives.
Diagnostic Metrics: These sorts of metrics help us to analyze a problem and understand the causes behind less than stellar results on a health metric. Diagnostic metrics often categorize some sort of event and report on the relative rates of occurrence of the various categories. Another sort of diagnostic metric may report on the performance of various processes that together contribute to the achievement of results measured by a health metric.
Let’s take an example to illustrate the differences here.
Progress Metrics: Project A is nearing completion, and the project manager is reporting that it is almost exactly on track in terms of cost and schedule. Life is good! No corrective action needed, and the project completes on schedule in the next few months.
Health Metrics: A short while later, the organization responsible for Project A performs its semiannual satisfaction survey of its customers. Despite the successful completion of Project A, its customers are less satisfied with the organization than last year! While all of the peer organizations’ scores have gone up! And the annual goals have not been met!
The organization is not happy. Stern memos are issued. Everyone is cranky (which makes customers even unhappier, although this won’t show up until the next cycle of customer satisfaction surveys).
Diagnostic Metrics: A followup survey is done with key customers, asking them to rate their levels of satisfaction with various aspects of the organization’s performance, and asking for comments. It turns out that they are unhappy with the organization because they always seem to deliver less than they had originally promised, and deliver it later than needed. (Of course, since Project A’s plans were rebaselined three times, and the project was descoped along the way, those rosy progress metrics didn’t do much to make the customers happy!)
All three types of metrics have legitimate uses. However, they are used for different reasons, so lumping them together often causes confusion. If you want to know how well your organization is performing, then looking simply at progress measures won’t be helpful. If you want to understand how you can improve, then health metrics alone won’t offer much help, and diagnostic metrics may be in order.
April 3, 2011
Next: Human Nature