Assessing IT Performance

blowing in paper bag to avoid hyperventilatingI received an email from a friend today asking if I would comment on  How to Fearlessly Grade a College Tech Department’s Performance, the current installment of Tech Therapy blog by Jeff Young on the Chronicle. This segment features an interview with Sandra Smith, Co-CIO at St. Lawrence University who discussed IT performance management. I enjoyed the 20 minute segment as I usually do.

IT Performance Management

The Tech Therapy segment left me wondering why the thought of setting up an IT performance management model and conducting and IT performance assessment would create so much anxiety you would need a paper bag. All of the other thoughts and reactions I had from the talk generally fall into one of these three themes:

  1. Benchmarking is irrelevant
  2. Best practice is only what you can get done
  3. Playing not to lose

Benchmarking is Irrelevant

During the discussion it seemed everyone agreed how odd it is that in higher education where our core function centers on measuring performance we struggle internally to assess our own IT performance including in IT departments. Like in so many other areas of higher education, we gravitate to comparing ourselves to “cohorts” or “peers” and the use of benchmarks.

Assessing one’s own performance can be scary. And as was pointed out in the segment, the fears of not knowing if your job is on the line or not or if the results will cause embarrassment are real and can be enough to keep CIO’s from doing anything on IT performance management. In these situations benchmarking offers the illusion of doing something but it is almost entirely irrelevant.

The goal of any IT performance management model ought to be continuous improvement. That’s the entire idea behind all of the IT service management and IT control frameworks. Control through active management supported by metrics. It doesn’t matter if you subscribe to ITIL, ISO/IEC, CoBIT, ITGI, Six Sigma, CMMI, or any one of the maturity models.

One example that came up was related to the benefits of the EDUCAUSE Core Data Service to identify if an institution may not have sufficient bandwidth. But comparing your institutions’ bandwidth to the aggregated data from a survey taken months earlier from a few hundred schools is absolutely irrelevant.

If you want to know if you have adequate network capacity and utilization – instrument it, track it, and decide if you need more or even if you have possibly over provisioned. Managing bandwidth capacity and utilization is a well establish process relying on continuous monitoring and generally accepted metrics from our industry just as with every other aspect of our standard services and functions. How much pipe others say they have in the CDS won’t ensure you have adequate user experience or not.

Best Practice is Only What You Can Achieve

The pursuit of “best practice” may be a real time waster and I have always said best practice is only what you can get done. This is a pragmatic view based on the reality of having helped lots of organizations big and small. Factors such as culture have a huge effect on this as do so many other dynamics and variables.

This idea works closely with that offered on benchmarking. True, there is performance data out there for every aspect of IT operations, finance, service desk, support processes, and project performance. But there are so many variables most never consider. ITSM is clear on being a framework that you must adapt as do the others. Six Sigma and Lean are about improvement methodologies not standards. Just because you are at or above a particular metric doesn’t mean you can’t or won’t have to do better. This is the fundamental tenant you should build you IT performance management practices around.

Consider the IT organization model and financing of IT (centralized, decentralize, federated) is terribly relevant in any comparison. Public sector to private sector is also huge in any practices related to staffing and several performance and financial metrics. To illustrate consider a public sector, possibly union, IT worker may only have 1550 hours per year for productive work, while a private sector counterpart will work well over that, possibly more than 2080 hours in a year which would affect numerous best practice numbers.

But I suppose the main thought I had in this area is that you can easily miss the big picture of IT performance management when trying to achieve a best practice. A simple example comes from service desk whether you subscribe to HDI or ITIL terms where it is all too easy to manage only on average speed to answer (ASA), average talk time (ATT) or first call resolution (FCR) rate. What most people miss is the balance between these metrics and others like satisfaction and true first call resolution rate. Efforts to improve average talk time can push customer satisfaction down and reduce true first call resolution. All work on performance improvement requires a systems understanding of the entire process and how each metric works with the others so when you begin to move on one you can also hold the others steady.

Playing Not to Lose

A great deal of what I heard today had a subtle message of a mindset of playing not to lose instead of playing to win. I am sure you can all think of examples from sports that illustrate this. Mine comes from growing up a Chicago Bears fan and having to watch them go into a prevent defense in the 4th quarter only to give up the lead and lose.

Other examples of playing it safe exist all around us. In this case the sentiment of just not doing an annual assessment or even setting up an IT performance management system is a way of not knowing how bad IT performance might actually be and by extension having to deal with the issues. A head in the sand management style.

It seems to me we have a tremendous obligation to deliver the best service we can manage to provide to our users and our customers – the students. I know many people who don’t like the idea of continuous improvement as a strategy and instead adopt one of pursuing “cost-optimization”. Fair enough, but you still have to track performance against the cost.

But if there is anything that makes me crazy and pushes my buttons it’s hearing someone say “you should have seen how bad it was 2 years ago”. For me, hearing that is just an excuse for why it’s not better now. I am much more comfortable setting targets, measuring key indicators and tracking performance towards a goal making needed adjustments along the way instead of anecdotes about the bad old days. I believe it is better to set a stretch goal and come up short than not to have a goal at all?

Another aspect of playing to lose is the idea of having to start small. Regardless of how you define it (re: low hanging fruit) starting small means it will take you years before anything meaningful will be noticeable to your users and customers. Maybe that is the strategy.

Worse yet, while you are starting small, your customer may be getting ready to go direct to a third party or the cloud with their needs. Just like you can’t imagine deploying a new piece of infrastructure without the ability to instrument and manage it, the rest of your functions have to have that visibility too. That is why making time to get visibility into your total performance ought to be priority #1 in every IT performance management system.

Getting Started

I know Tech Therapy is supposed to a light dialogue on a range of topics. I realize others like me take it in while eating my lunch at their desk. It is supposed to be a gentle nudge for all of us to move forward perhaps a bit more today than we had planned. I only wish today’s discussion shared a higher expectation for our profession. Expectations to achieve greater things for ourselves and the students and faculty we serve.

Sometimes all that is needed is a nudge from a firmer hand and to know what is truly possible. On metrics, process improvement, IT control frameworks, and continuous improvement but also on how to do your own assessment of these areas throughout the year so you can avoid having to bring in an outsider which only raises the fears of the assessment. IT management is a profession with standards and methods which can be leveraged to address anything an assessment uncovers.

So if you really want to grade your IT department I will dig around for some materials to post so you can start your own journey of assessment and improvement. There is also a ton of stuff on the web that you can use yourself without any trouble. Either way, you will be able to get started with more than a small effort without any anxiety from bringing in an outsider.

This entry was posted in CIO Job, IT Performance Management and tagged , , . Bookmark the permalink.

2 Responses to Assessing IT Performance

  1. Pingback: IT Performance Assessment You Can Do Over Pizza and Pop | The Higher ED CIO

  2. I have a lot of respect when I see articles by other people who will be more of an expert as compared with me on this topic. The audience might think that looks effortless, but there’s so much going on behind the scenes. Your work is a result of analysis and knowledge by you and you were very good enough to share these with people like me. Thanks a lot.

Comments are closed.