Wednesday, March 23, 2011

Some thoughts about good metrics...

If you are like me then you must be facing some measurement challenges every now and then at work. There are some general rules that one needs to be aware of. Below is a short summary of some of those measurement rules, they mainly come from three sources that I highly recommend:
Let's start with defining "Measurement". Doug Hubbard offers a simple definition for measurement "A set of observations that reduce uncertainty where the results is expressed as a quantity." (he looks at it as an information issue, read the book if that sounds interesting to you).
So basically if your observation is expressed as a quantity and you know after it more than you did before then we can call this observation a measurement.  The additional point here is that the benefit of the measurement should be higher than the cost of making the measurement; else it's not of much value. 

So what are the characteristics of good metrics?
  • Complete and Relevant: Good metrics are connected to important goals. Good metrics are the translation of goals expressed in English to math. The measure must accurately cover your needs to be useful. In some cases you might need multiple measurements to support your business needs. One metric alone might not be sufficient for drawing conclusions and decision making. Also remember that everything changes; your once relevant metrics might not be as relevant in the future.  
  • Consistent: Good metrics always go into on direction if things improve and the other if they deteriorate. The measure will always be the same no matter who carries out the measurement and records it. You can always count on the numbers. It's not a subjective exercise. If feasible the measurement should be automated.
  • Expressed as a cardinal number or percentage, not with qualitative labels like “High”, “medium”, and “low”
  • Expressed using at least one unit of measure, such as “defects”, “hours”, or “dollars”
  • Communication: The results need to be communicated. Measurements are typically shared with others. It should be easily recognizable what the metric means. When the metric drops/rises to unexpected levels the source of the problem and the necessary actions are clear to your audience.
  • Presentation: The presentation of measurements can be as important as the measurement itself. Spend some time on data visualization and always keep it simple.

One last tip, question conventional wisdom and metrics that are suggested by others to you. Validate them using the above rules. Every now and then you will come across a metric mentioned in a reputable source that just doesn’t stand the validation test.

No comments:

Post a Comment