Home » Time Capsule » Currently Reading:

Time Capsule: 2006 Product Rankings – Pay Some Attention, But Not Too Much

December 23, 2011 Time Capsule No Comments

I wrote weekly editorials for a boutique industry newsletter for several years, anxious for both audience and income. I learned a lot about coming up with ideas for the weekly grind, trying to be simultaneously opinionated and entertaining in a few hundred words, and not sleeping much because I was working all the time. They’re fun to read as a look back at what was important then (and often still important now).

I wrote this piece in December 2006.

2006 Product Rankings – Pay Some Attention, But Not Too Much
By Mr. HIStalk


The 2006 Best in KLAS Awards were just announced. I know a lot of folks wait anxiously for those, especially vendors.

KLAS has its critics, of which I’m occasionally one. I like the idea of surveying customers, but having taken KLAS surveys myself, I’ve seen first-hand how poorly designed some of their questions are. A respondent can’t complete the survey without in-depth knowledge of product functionality, support, documentation, executive relationships, hardware, and so on.

In other words, either the person cares so much that they carefully enlist the assistance of 3-10 colleagues, or they just wing the damn thing so they can get back to work instead of wasting time on yet another hard-to-finish survey. You guess which. At least KLAS sometimes follows up by phone.

All of that data collection and interpretation imprecision is masked by a numbing array of graphs, charts, and tables in the final product. No one pays attention to those, recognizing them as a way of padding out the sometimes-skimpy data to make it more impressive and convincing.

The scores tend to wander a lot from one report to the next, which KLAS attributes to swings in vendor or product performance. It’s much more likely to have been caused by the imprecision of the survey methodology. Do you really believe that Meditech’s client-server EMR was improved so much that it earned itself a move from sixth in an eight-horse race all the way up to second place in just one year? Me neither. KLAS marks it as having a low confidence level, so how do you interpret its score?

Still, I’d definitely pay attention to the top-ranked product and be careful with the last-place finishers. I wouldn’t try to overanalyze products in between, especially when the scores are close. Read the customer comments instead, the best part of the KLAS reports if you ask me (although you don’t know what kind of organizations and employees are being represented.)

The most discouraging point is that no vendor does everything well, if you believe the scores. If you look at KLAS’s 15 main general solutions categories, you’d find 13 top-ranked vendors. If you’re a best-of-breed shop, you’ll end up with a lot of interfaced systems if you chase the winners. If you’re a single-vendor organization, some of your departments are going to be stuck with systems far short of being the best. And of course, in the next survey, they may all shuffle around anyway.

My takeaway is this: somebody has to be last, and if the scores are tight, it may not mean much. On the other hand, why can’t a #1 ranked vendor in one product area use that knowledge to excel in the others? Or, do vendors selectively invest in some strategic lines and allow the others to languish? What does the #1 vendor know or do that everyone else doesn’t?

You don’t have to be a KLAS subscriber to see who’s #1. Just check out the HIMSS booth walls and vendor marketing material. You will need to pony up, though, to see who earned worst in KLAS and which vendors did well in important categories like “would you buy it gain.” That’s where the subscription is worth it. Use it like Consumer Reports for car-shopping – at the beginning and end of your selection process. The work in between is on your own.

View/Print Text Only View/Print Text Only

HIStalk Featured Sponsors


Subscribe to Updates



Text Ads

Report News and Rumors

No title

Anonymous online form
Rumor line: 801.HIT.NEWS



Vince Ciotti’s HIS-tory of Healthcare IT

Founding Sponsors


Platinum Sponsors






















































Gold Sponsors
















Reader Comments

  • Math: Dittoing on how awesome Zenni is, I've been using them for a couple of years. Most people don't realize how much of an ...
  • Code Jockey: Gotta love it when the kids show up..... After a number of years of almost constant travel and well over 700+ flights...
  • Kermit: My family's been happily using Zenni for years. Never a problem with the quality of their products. My last order was fo...
  • Cosmos: During an inpatient stay at a Boston area teaching hospital, my wife was approached by a research assistant about enroll...
  • AlphaCodeJockey: Finally, CodeJockey has a incentive to play up issues with developing custom solutions for Epic's platform as improvemen...
  • DeltaCodeJockey: I'd also point out that Epic insiders are probably right to be touchy about this topic as they get flack publicly and li...
  • Gamma Code Jockey: Code Jockey's example of an IT form that ties into Epic's system is wrong though - you are allowed to do that and you ar...
  • Bravo Code Jockey: Code Jockey - thanks a tonne for persisting and for your rational and informed responses to a lot of name calling and mi...
  • Bill O'Toole: Confirming first hand that Cape Cod Hospital was MEDITECH's first customer. My father, the Chief of Pathology, saw the i...
  • FormerCIO: Re JAMA article. How will access to my EHR data help me 'shop for high value health care services' and 'avoid the need t...

Sponsor Quick Links