5 Steps to Transparent Metrics

Measuring and reporting on IT efficiency and effectiveness is critical. The ITIL mentions Continuous Service Improvement Programs (CSIP), Goals, Critical Success Factors (CSFs) a.k.a. Key Success Factors, and Key Performance Indicators (KPIs) as the means to measure success. In fact, ITIL V3 even devotes an entire phase of the IT Service Lifecycle to Continual Service Improvement (CSI)!

A CSIP establishes goals based on business drivers and stakeholders. CSFs are important performance aspects of CSIP goals. KPIs measure CSFs to show achievement of CSIP objectives. Together CSFs and KPIs track IT performance.

The ITIL suggests dozens of KPIs; CobiT and ISO-20000 suggest hundreds more. [See ‘Get Ready for ISO 20000 Certification’ DITY Vol.2 Iss.3 for more on ISO 20000.] However, every CSIP has unique KPI needs that require mixing people, process, and products into performance metrics, and then reporting on them in business terms. Called IT transparency, this is not an easy task.

Fortunately, there is an industry standard for creating effective KPI metrics linked to goals — the Goal Question Metric (GQM) method. GQM offers us a solution for creating transparent KPI metrics. Following we examine GQM.

Fundamentally, KPIs measure progress toward goals as reflected in CSFs. KPIs are then quantifiable measurements, or metrics. The Goal Question Metric (GQM) method arose from work done in software engineering. GQM derives metrics from questions about goals. GQM has some powerful benefits when applied to IT Service Management:

• simple, form-driven application
• easy to use in groups
• works with all processes
• produces metrics customized to the CSIP
• delivers powerful insight into organizational and process structure
• reflects the viewpoints of the staff (IT, customer, stakeholder) involved

GQM requires no investment in tools or software. It is a participative and team-building program that helps drive values up and down the organization. GQM is useful in creating new KPIs, but it is also very useful in clarifying your existing KPIs. GQM is easiest to grasp when using a real example, so, here is an example, loosely based on examples in the ITIL:

• The CSIP has as a goal of "providing best customer response among all major competitors."
• One of the IT CSFs that came from this goal was "quickly resolve incidents."
• A potential KPI offered in the ITIL for such a CSF is "Percentage reduction in average time to respond to a call for assistance."

This KPI needs clarification and is what I refer to as a “Dr. Feelgood” KPI -- it sounds nice, no one could argue with it, it seems to make sense and “fit” the CSF. However, how would you actually measure this KPI? To whom does it apply? What does it actually measure? It is almost impossible to determine the “who, what, when, where and why” given this KPI. This is where GQM is very powerful. GQM uses a standard form to define the metric and to whom or what it applies very precisely. GQM operates at three levels:

• Conceptual/Goal: a set of objectives that represent various viewpoints relative to a specific environment. With regard to ITSM, goals are the objectives and milestones of the CSIP.
• Operational/Question: a set of questions about the goal that focuses characterizing the assessment or achievement of a specific goal. The answers to these questions determine goal achievement. With regard to ITSM, the questions derive from the CSFs.
• Quantitative/Metric: a set of measurements that answer the questions. With regard to ITSM, the metrics are KPIs.

The five questions or parts of the GQM method are:

• Object: What is the Object (product or process) under measurement?
• Purpose: What is the motivation behind the analysis of the Object?
• Focus: Which quality attribute of the Object is under study?
• Environment: Under what context is the measurement of the Object attribute to occur?
• Viewpoint: Whose perspective do measurements of the Object represent?

Let us use GQM to break down our (unclear) KPI into the required parts:

GQM Part	GQM Question	KPI Part
Object	What is the Object (product, process, service, etc.) under analysis?	??
Purpose	What is the motivation behind the analysis of the Object?	Reducing the average time required
Focus	Which quality attribute of the Object is under analysis?	Responding to a call for assistance
Environment	Under what context is the analysis to occur?	??
Viewpoint	Whose perspective does the analysis of the Object reflect?	??

Asking these five simple questions quickly shows that our potential KPI of “Percentage reduction in average time to respond to a call for assistance” is missing the key information that actually makes it useful:

• What is the object -- Staff member? Procedure effectiveness? Knowledge base? System response time?
• What is the purpose -- To reduce the average time to respond or understand the reasons for the time it takes to respond?
• What exactly are we going to measure -- Response time? Hold time?
• What is the environment -- the Service Desk? Incident Management?
• What is the viewpoint -- the User? IT Staff member?

You begin to see the value of GQM now. Using GQM we can clarify the potential KPI into a KPI that is actually useful. We have to go back and ask more questions in order to clarify the KPI. For example:

GQM Part	GQM Question	KPI Part
Object	Analyze:	The Automated Call Distribution system
Purpose	For the purpose of:	Reducing the average time required
Focus	With respect to:	Time spent in the queue
Environment	In the context of:	Service Desk during normal business hours
Viewpoint	From the viewpoint of:	A Business User calling the Service Desk

We then re-write the KPI. Here is the original KPI and CSF:

CSF: Quickly Resolve Incidents.

KPI: Percentage reduction in average time to respond to a call for assistance.

Here is the clarified KPI and CSF:

CSF: Quickly Resolve Incidents.

KPI: Percentage reduction in the average time a business user waits in the ACD queue when calling the Service Desk during normal business hours.

You can clearly see the value of GQM now. Notice how much more clearly the re-written KPI now appears. Notice how it clearly indicates the “who, what, when, where and why” for the KPI. As an IT manager, you know what data to gather, and when, where and why to obtain the data. If you were the customer (or anyone else inside or outside of IT) you would know what the metric meant by simply reading it. This is an important concept in achieving the IT transparency required of IT/business alignment.

Also, notice that the clarified KPI only addresses a portion of the CSF! GQM will often result in many new KPIs. Now that we have re-written a KPI, you can also use GQM to create entirely new CSFs and KPIs from CSIP goals.

In either case, some basic guidelines apply:

1. Make sure you have management commitment supporting the measurements and the evaluation of the measurements; this is naturally occurring if you drive GQM from the CSFs taken from the CSIP, and the CSIP truly represents stakeholder values and needs. However, metrics for the sake of metrics are a very real and negative consequence of failure to obtain management commitment before you begin with GQM.
2. Involve stakeholders in the GQM process; create a GQM team composed of IT, customer, and other stakeholders in this process. The GQM team should both assist in the creation, and the analysis, of the KPI data.
3. GQM measurements are not organizational goals or project goals in themselves; they are measurements of goals and objectives. Be careful not to confuse the pursuit of KPI attainment with the goals themselves.
4. Always consider the context of the viewpoint; the GQM team may identify many KPIs for a single CSF, each from a different context or viewpoint. Normally one CSF will have several to many KPIs.
5. Develop metrics appropriate to the object; be careful to measure attributes specific to the goal and when analyzing them stay focused on the object. Every KPI reflects a singular aspect of a goal -- do not try to "read into" the KPI for information beyond its clearly stated purpose. If you need more data, create a specific KPI.
6. Involve Project Management and make GQM a regular part of IT project management. Helping establish the measurements of projects gives you a better chance of actually meeting those objectives!
7. Get training in GQM before moving forward!

SUMMARY:

One of the most difficult issues you face is to measure improvement from ITIL. The only way that works is transparency -- expressing IT results in business terms, linked to stakeholders. GQM makes this easy. GQM ties into the ITIL CSIP/CSF/KPI model very nicely, and provides a straightforward means to both identify improvement goals and measure their performance. You will also find that the GQM process itself helps identify process problems that you can address (and measure) using additional GQM derived KPIs! Finally, communicate the results of your clarified KPIs to IT and customers, this is a tangible step forward on the path toward Business/IT alignment.