Wednesday, March 23, 2011

Some thoughts about good metrics...

If you are like me then you must be facing some measurement challenges every now and then at work. There are some general rules that one needs to be aware of. Below is a short summary of some of those measurement rules, they mainly come from three sources that I highly recommend:
Let's start with defining "Measurement". Doug Hubbard offers a simple definition for measurement "A set of observations that reduce uncertainty where the results is expressed as a quantity." (he looks at it as an information issue, read the book if that sounds interesting to you).
So basically if your observation is expressed as a quantity and you know after it more than you did before then we can call this observation a measurement.  The additional point here is that the benefit of the measurement should be higher than the cost of making the measurement; else it's not of much value. 

So what are the characteristics of good metrics?
  • Complete and Relevant: Good metrics are connected to important goals. Good metrics are the translation of goals expressed in English to math. The measure must accurately cover your needs to be useful. In some cases you might need multiple measurements to support your business needs. One metric alone might not be sufficient for drawing conclusions and decision making. Also remember that everything changes; your once relevant metrics might not be as relevant in the future.  
  • Consistent: Good metrics always go into on direction if things improve and the other if they deteriorate. The measure will always be the same no matter who carries out the measurement and records it. You can always count on the numbers. It's not a subjective exercise. If feasible the measurement should be automated.
  • Expressed as a cardinal number or percentage, not with qualitative labels like “High”, “medium”, and “low”
  • Expressed using at least one unit of measure, such as “defects”, “hours”, or “dollars”
  • Communication: The results need to be communicated. Measurements are typically shared with others. It should be easily recognizable what the metric means. When the metric drops/rises to unexpected levels the source of the problem and the necessary actions are clear to your audience.
  • Presentation: The presentation of measurements can be as important as the measurement itself. Spend some time on data visualization and always keep it simple.

One last tip, question conventional wisdom and metrics that are suggested by others to you. Validate them using the above rules. Every now and then you will come across a metric mentioned in a reputable source that just doesn’t stand the validation test.

Friday, March 18, 2011

Security Firm RSA got hacked...what does that mean to the rest of us?

By now anyone following information security must have heard about the RSA hack. If not go and google it for details. What exactly happened is not clear, RSA is talking about some sophisticated attack, but I guess they would say that no matter what. It doesn't really matter.

So what does that mean to the rest of us:
  • If you are going to get hacked or not seems to be only a function of wether you are an attractive target or not. All other variables (attack vectors, risk, vulnerability...) seem to be just noise. It comes down to incentives and economics.
  • I still meet too many people whose eyes sparkle when they get the chance to talk about the latest security tools they implemented. Typically those start to fall apart a few months after the implementation. But the lesson here is that as much as you depend on security tools they are only as good as the people that designed them, the implementation and maintenance. Additionally they also can be your Achilles heel. Besides the complexity that they add, every now an then we find security tools that can somehow be exploited.
  • Instead of running after the latest security hypes, make sure you cover the basics:  least privilege, segregation of duties, change management, zoning, incident response, risk management, recovery, patch management, awareness, social engineering, default deny, defense in depth.....
And this concludes my first post.