Survey Says! Effectiveness of Technology Investments

Technology Investments Survey SaysI have been stewing over several higher education survey reports lately to the point of it becoming an obsession. So I have decided to try convert my obsession into action with the hopes of it relieving my own stress but also to address what I see as a major issue.

And that is, there is a lack of information in higher education on products and vendors essential to effective decision making on IT investments that is normally found in other industries.

What’s more troubling to me is it seems there is a core set of questions we just don’t seem to want to know the answers to – and so we never ask.

The Question of Value

There is really only one question everyone tries to answer when making an IT investment decision.

“Which of the products available will sufficiently meet the need at a price (re: total cost) I can afford?”

This is the age old question of which product offers the greatest value.

Many recognize the idea of value as the balance of cost vs quality. When you think in terms of value being the trade-off between cost vs quality it can be easier to focus on only buying just enough product to be good enough while still being affordable.

There is no one formula for value although there are clear methods for determining total cost of ownership (TCO) and comparing one IT investment option with others.

So we develop models for evaluating IT investment options based on value that pull in other factors based on the business case driving the proposal. That’s what the book Making Technology Investments Profitable is all about.

That’s why you incorporate factors like the customer satisfaction ratings of vendors or at least contact references. It’s also why you want to know about benchmark performance levels of products across a vendor’s customer base.

Survey Says!

Unfortunately, higher education lacks this data. Sure there are plenty of reports generated each year which focus in some way on technology issues in higher education. But what do those reports really offer? Do they inform your IT investment value decisions?

Nearly every report relies on a survey method to develop its data. The surveys aimed most directly at IT are focused mostly at the technology which exacerbates the longstanding stereotype of IT professionals and perpetuates the issues of value.

In most of these reports it is mostly the speeds & feeds of IT that receive the attention with no consideration given to actual results or the effectiveness of specific technologies, products, or strategies.

Then there are the three reports from Inside Higher Ed giving me the most trouble right now.

Each of these reports develops important insights into just how institutional leaders see IT investments effectiveness in solving  institutional strategies, educational goals, and financial challenges.

IT Effectiveness in Higher Education

I think the data in these reports is awful. Yet, another year goes by and barely a peep from the IT side of the higher education community. No conclusions in the reports as to causation or ideas for addressing the issues – and these are issues.

Let’s look at the key data points regarding “Effectiveness of Campus Investments in Information Technology” (percentage reporting “very effective” scores of 6 or 7; scale of 1=not effective, 7=very effective) in the Inside Higher Ed reports.

Survey Item




Administrative information systems and operations 48.0 33.4
Data analysis and managerial analytics 41.8 28.6
Using financial data to inform campus decision-making 39.4
Using technology to enhance the institution’s mission and finances 22.8

Unanswered Questions on the Value of IT

Now considering these data points I immediately want to know why almost no IT strategic plans are addressing these issues. If you disagree, show me.

But I really want to know what vendors and  software platforms the institutions are using relative to the effectiveness numbers.

I want to know if institutions running Moodle have a different view than those running Sakai, BlackBoard, or D2L. That’s a question of value after all.

I am curious if there is a difference between COTS (commercial off the shelf software) and OSS (open source software) or those running their LMS on premise (BlakcBoard, Moodle or Sakai) versus hosted (BlackBoard, Moodlerooms, or rSmart).

I want to know if it makes a difference where the institution is on the eLearning adoption curve. Do early adopters feel better about their technology investments than those just getting started? That might help us learn if things really do get better or if cutting your losses makes more sense.

Similarly, I want to know which SIS or ERP (G/L) software package the institutions are running and what BI (business intelligence) tool is being used. After all, it seems like the whole point of running information systems is to get at the information to support management decision making to produce performance results.

So it seems like it might be useful to know if those running Campus Management, Ellucian (Datatel or SunGard), Oracle, Jenzabar, RJM, Three Rivers, or TopSchool institutions have differing effectiveness ratings. It seems that might be at the heart of the issue of how hard it is to get at your data and convert it into information and actionable insights.

Knowing effectiveness by what BI and reporting tools institutions are using, if any, is also just as important. And so too would knowing if self-hosting makes a difference versus using a shared service model of a consortium or central system or even a vendor hosted approach because it changes your view of value in the cost vs quality equation.

What really surprises me most is the CFO survey data. When you consider that many CFO’s oversee IT you’d think the effectiveness ratings would be higher or we’d see CFO’s across the country lighting their hair on fire to change things.

But then in all three survey reports there is very little to suggest institutional leaders see IT or technology investments as a part of the solution to their current challenges and initiatives. That runs squarely against the case being made by the IT side of the issues.

And that should be troubling to every CIO.

Call to Action

It is not enough to simply complain about an issue. It just feels like I should either do something to develop broader insights and answer these questions or just shut up about them. Unfortunately the same ego and arrogance that makes me think I see something here also keeps me from just accepting it.

So I am considering ways to go after some of this data and add to it some analysis and opinions so there is actionable information for decision making. There are some hurdles to overcome but they are not impossible and should be fun to work on.

Meanwhile, what questions would you want to see answered about vendors or products? Put them into the comments or drop me an email so we can see if we can answer them.

This entry was posted in CIO Job, IT Strategy and tagged , , , , . Bookmark the permalink.

One Response to Survey Says! Effectiveness of Technology Investments

  1. Pingback: EDUCAUSE Learning Initiative Thematic Framework

Comments are closed.