Time Capsule: 2006 Product Rankings – Pay Some Attention, But Not Too Much

I wrote weekly editorials for a boutique industry newsletter for several years, anxious for both audience and income. I learned a lot about coming up with ideas for the weekly grind, trying to be simultaneously opinionated and entertaining in a few hundred words, and not sleeping much because I was working all the time. They’re fun to read as a look back at what was important then (and often still important now).

I wrote this piece in December 2006.

2006 Product Rankings – Pay Some Attention, But Not Too Much
By Mr. HIStalk

The 2006 Best in KLAS Awards were just announced. I know a lot of folks wait anxiously for those, especially vendors.

KLAS has its critics, of which I’m occasionally one. I like the idea of surveying customers, but having taken KLAS surveys myself, I’ve seen first-hand how poorly designed some of their questions are. A respondent can’t complete the survey without in-depth knowledge of product functionality, support, documentation, executive relationships, hardware, and so on.

In other words, either the person cares so much that they carefully enlist the assistance of 3-10 colleagues, or they just wing the damn thing so they can get back to work instead of wasting time on yet another hard-to-finish survey. You guess which. At least KLAS sometimes follows up by phone.

All of that data collection and interpretation imprecision is masked by a numbing array of graphs, charts, and tables in the final product. No one pays attention to those, recognizing them as a way of padding out the sometimes-skimpy data to make it more impressive and convincing.

The scores tend to wander a lot from one report to the next, which KLAS attributes to swings in vendor or product performance. It’s much more likely to have been caused by the imprecision of the survey methodology. Do you really believe that Meditech’s client-server EMR was improved so much that it earned itself a move from sixth in an eight-horse race all the way up to second place in just one year? Me neither. KLAS marks it as having a low confidence level, so how do you interpret its score?

Still, I’d definitely pay attention to the top-ranked product and be careful with the last-place finishers. I wouldn’t try to overanalyze products in between, especially when the scores are close. Read the customer comments instead, the best part of the KLAS reports if you ask me (although you don’t know what kind of organizations and employees are being represented.)

The most discouraging point is that no vendor does everything well, if you believe the scores. If you look at KLAS’s 15 main general solutions categories, you’d find 13 top-ranked vendors. If you’re a best-of-breed shop, you’ll end up with a lot of interfaced systems if you chase the winners. If you’re a single-vendor organization, some of your departments are going to be stuck with systems far short of being the best. And of course, in the next survey, they may all shuffle around anyway.

My takeaway is this: somebody has to be last, and if the scores are tight, it may not mean much. On the other hand, why can’t a #1 ranked vendor in one product area use that knowledge to excel in the others? Or, do vendors selectively invest in some strategic lines and allow the others to languish? What does the #1 vendor know or do that everyone else doesn’t?

You don’t have to be a KLAS subscriber to see who’s #1. Just check out the HIMSS booth walls and vendor marketing material. You will need to pony up, though, to see who earned worst in KLAS and which vendors did well in important categories like “would you buy it gain.” That’s where the subscription is worth it. Use it like Consumer Reports for car-shopping – at the beginning and end of your selection process. The work in between is on your own.