Everyone knows metrics measure things.  What most do not understand are the dirty little truths about metrics -- what gets measured is what gets done, and metrics drive both good and bad behavior.  Put another way, people do what you pay them to do.

By Hank Marquis

The IT Infrastructure Library™ or ITIL® mentions metrics frequently -- Critical Success Factors (CSF) and Key Performance Indicators (KPI).

The lesson is simple -- metrics must derive from, and align with, business goals and strategies.  Metric selection should occur only after understanding the needs the metric addresses. 

However, most managers do it backwards.  They have metrics they want to see, and design processes around them.  This is usually a sure path to failure.  This common problem is not unique to IT, and examples of misguided metrics abound inside and outside of IT. 

A classic example used by MIT is that of Continental Airlines recovering from bankruptcy in the 1990’s.  Continental had to cut costs, and since fuel costs were high management created a metric to track its usage.  They used this metric to reward pilots for reducing fuel consumption.  As expected, pilots acted to earn their reward -- they skimped on air conditioning and flew more slowly.  Management got what it seems they wanted, and fuel consumption fell.  Unexpectedly however, their customer satisfaction and on-time performance also fell, and their most valuable frequent flyer customers moved on to competitors. 

An IT example I recently came across was a metric about aging trouble tickets.  Management established a metric to monitor ticket age.  However, many tickets did not close within the established timeframe.  The Service Desk manager met the metric by closing and reopening the tickets -- in effect, faking the numbers.  But management got what they seemed to want -- a nice report showing that everything was fine. 

Clearly, in each of these cases, the true goal of management was not achieved.  In neither case did management really desire the outcome realized.  However, failure to follow a rigid model for the creation of metrics invariably results in failure, and a failure to lead is not a failure of the led. 

Following I describe the real purpose of metrics, and explain how to create or select metrics for IT that produce the results you really desire.

Misguided Metrics

Most managers believe they know what a metric is and how to use them, however, given the number of examples like the last two, it seems many really don't. 

One analogy I often use to explain the function of metrics is that of an automobile trip.  Using the trip analogy, the goal might be to get across the desert.  Goals are extending, and should grow an organization.  In this analogy a goal is the destination.  CSFs are things you have to do to achieve the goal, CSFs are attainable.  One CSF from our analogy might be to avoid overheating.  A KPI is a measure of some property of a CSF.  The KPI in this analogy might be water temperature, as expressed by a temperature gauge.  By watching the water temperature gauge we have an indication if we are likely to meet the requirement to avoid overheating, and if we don't overheat we stand a good chance of driving across the desert.

If we were not crossing the desert in a car then our CSF and KPI would probably be quite different.  For example, if we were flying a plane across the dessert instead of driving a car through it, we would probably also want to know about air speed and altitude, and water temperature might not be required. 

KPI metrics (like water temperature) derive from CSFs (like avoiding overheating).  CSFs derive from goals (like driving a car across the desert).  Goals derive from current process maturity and business/mission requirements.  Every goal will probably have unique CSF, which in turn probably have unique KPI.

The issue with metrics is that management often confuses process and performance, mixing up goals, CSFs and KPIs.  In effect, management asks staff to fly an airplane across the desert using altitude and air speed as metrics.  However, knowing they need to drive a car because (contrary to management assumptions) they don't actually have a plane, staff must creatively interpret data to fit into required management metrics.  We wind up with meaningless reports that deliver no relevant information.

Misguided metrics led Continental pilots to drive away their best customers -- and to do so under management orders!  Misguided metrics are also responsible for most of the reporting shenanigans going on right now in virtually every enterprise and service provider IT organization. 

Here are 7 dirty little truths about metrics...and 7 things you can do to clean up yours.

Truth #1

What gets measured IS what gets done.  Regardless of what you say or what you might consider to be common sense, people will do whatever they believe required to meet the metric.  Thus, metrics have to reflect a complete understanding of what you really want to get done.  Start with an understanding of the business or mission, then identify process customers and key stakeholder interests before selecting metrics.  The actual metric is one of the last things the process manager should settle upon.  Unfortunately, the most common reality is selection of metrics before establishing the process, or worse, choosing metrics because the tools currently in use offer them.

Truth #2

Metrics drive both good AND bad behavior.  People do what you pay them to do, so choose carefully.  People will do whatever they believe required to meet required metrics (see Truth #1) especially if there are performance rewards for attaining the metrics.  The metrics you choose will drive your organization, as they should.  However, failure to think outside of the immediate process or activity creates well meaning but poor metrics.  For example, a "punitive" focus on ticket age without "forgiveness" as to the mitigating reasons can result in fear of potential retribution of a "bad score" on a report.  The organization will act accordingly to avoid potential retribution.

Truth #3

Failure to align with Vital Business Functions (VBF) can lead you and your team astray.  Failure to align your metrics with VBF results in metrics that measure the wrong things for the wrong reasons.  Every single metric in IT needs to trace all the way back to what the ITIL calls a VBF.  In other words, nothing should be done in IT that is not part of a solution for something in the business.  You have to start with a mission statement for the process, work through goals and objectives and then create your metrics.  Only this process of aligning your process to the VBF, and then measuring to ensure the process meets the requirements of the VBF, is effective. 

Truth #4

Metrics do not get better with age -- they often become obsolete.  As the organization matures, metrics usually must change to keep pace.  Good metrics reflect the reality of the current process or activity.  As staff gains skills, you normally set new goals with new objectives.  This also means you must refine or replace your metrics (CSF and/or KPI) to properly reflect attainment of your new goals.  Unfortunately, most companies settle on metrics for the sake of metrics.  Senior management has a handful of “favorite metrics” they have “used for years.”  In these unfortunate organizations, instead of customer satisfaction, attainment of the metric becomes the pursuit of the entire organization.  Usually the result is as you might now expect -- failure.

Truth #5

The real purpose of metrics is to help you make better decisions.  The objective of a metric is to operate as a gauge, or indicator of process operation.  Metrics are to aid in making good decisions, they exist to identify gaps in skills or resources required to attain the goal.  Good metrics focus on the issues that affect the team, and that put attainment of the process goal or objective into jeopardy.    

Truth #6

Effective metrics do not measure people -- they measure teams and processes.  The real purpose of metrics is to help your make better decisions (see Truth #5) and to not hold people accountable, or punish them.  Most traditional (poor) IT metrics exist to keep track of people and find out who is responsible when something goes wrong.  Metrics become a "blame game" (see Truth #2.)  Good metrics focus on the attainment of process objectives and goals.  In this way, the focus of a good metric is on the team and accomplishing the mission.  Individual performance metrics tend to cause the measured person to focus on doing what they think management wants, as reflected in the metric (see Truth #1.)

Truth #7

Good metrics help optimize the performance of the entire organization.  Metrics should be accurate, timely, and actionable.  You need sound metrics at all levels to prevent metrics that report nothing relevant and do not contribute to decision making.  Think of your metrics reporting as an opportunity to learn about your process.  Keep the list of metrics small, and focused.  Get team buy-in for team level metrics.  With continuous evaluation and re-evaluation with your team, you can optimize your process.  When multiple or interdependent processes follow this model the results are amazing.

Summary

To summarize, many metrics in use in IT today are at best worthless, and at worst counterproductive.  The following represents a simple checklist for creating metrics that matter:

• Align with Vital Business Functions.  Regardless of the IT activity, you need to make sure your metrics tells you something about the VBF that depends on what you are measuring.
   

• Keep it simple.  A common problem manager’s face is overloading a metric.  That is, trying to get a single metric to report more than one thing.  If you want to track more than one thing, create a metric for each.  Keep the metric simple and easy to understand.  If it is too hard to determine the metrics people often fake the data or the entire report.
   

• Good enough is perfect.  Do not waste time polishing your metrics.  Instead, select metrics that are easy to track, and easy to understand.  Complicated or overloaded metrics often require excessive work, usually confuse people, and do not get used.  Use a tool like the Goal Question Metric (GQM) model to clarify your metrics. [Please see '5 Steps to Transparent Metrics' for more on GQM.]
   

• Use metrics as indicators.  Key Performance Indicators are metrics!  A KPI does not troubleshoot anything, but rather the KPI indicates something is amiss.  A KPI normally does not track or show work done.  Satisfying several KPI normally means satisfying the related CSF.  The KPI is an indicator, a metric, designed to alert you that CSF attainment might be in jeopardy, that is all.  A good metric (KPI) is jut an excellent indicator of the likelihood of attaining a CSF.
   

• A few good metrics.  Too many metrics, even if they are effective, can overwhelm a team.  For any process 3 to 6 CSF is usually all that is required.  Each CSF might have 1 to 3 KPI.  This means most teams and individuals might have just 2-5 metrics related to their activities or process.  Any more and either the metrics won't get reported, or the data gets faked.  Too many metrics transforms an organization into a reporting factory -- focusing on the wrong things for the wrong reasons.  In either case, the usefulness of the metric is compromised. 
   

• Beware the trap of metrics.  Failure to follow these guidelines invariably results in process problems.  Look around your current organization.  How many reports are actually used to help make decisions?  Don't assume, ask around, you will probably be surprised.  Its not uncommon to discover reports written for group "A" are not used by group "A".  In many cases, group "A" thinks the report is for group "B" and vice versa.  The result is work without reward.  Discontinue reports no one actually uses.  Review each metric as well, and if they do not tell you something about the efficiency, effectiveness, economy, or equity of the process, discontinue or refine it.  It is all too easy to fall into the trap of metrics for the sake of metrics.

Where to go from here:

Related articles:

• 5 Steps to Transparent Metrics

Entire Contents © 2006 itSM Solutions LLC.  All Rights Reserved.