HIStalk Advisory Panel: How Do You Use Information from KLAS?

The HIStalk Advisory Panel is a group of hospital CIOs, hospital CMIOs, practicing physicians, and a few vendor executives who have volunteered to provide their thoughts on topical industry issues. I’ll seek their input every month or so on an important news developments and also ask the non-vendor members about their recent experience with vendors. E-mail me to suggest an issue for their consideration.

If you work for a hospital or practice, you are welcome to join the panel. I am grateful to the HIStalk Advisory Panel members for their help in making HIStalk better.

This month’s question: How do you use KLAS reports or scores to choose and monitor your vendors?

Generally Negative Comments

• I place almost no value in the KLAS scores. Years ago I took a class on research methods and the professor used them as an example of bad methodology. What is great about them is they know all the products from all the vendors, so when I’m looking for that niche departmental system, I can go to them for a list of vendors.
• (from a vendor employee) I believe that KLAS has a very flawed system, which has been brought to their attention time and time again with no changes. There is inaccurate information, and when brought to their attention, no changes are made. We are a vendor, and the information they continue to have on us is actually so inaccurate that it’s ridiculous. They list us as "small volumes" and every company but two on the list that is ranked (we are not because of this "small volumes" designation) is much smaller than we are. They will not correct it, so we have decided that it is not worth the hassle to continue to correct them, only to have them continue on as previous. On speaking with customers, we have been told that they have run into the same things in all categories and no longer give any weight to the rankings. A few even think that it is possible to pay for your ranking and rating.
• We use KLAS reports (if available) to supplement MAJOR capital purchases. Most of the reports are too expensive to justify unless the expected purchase is one in which we have no experience and is a major capital purchase.
• I review KLAS reports, but I do not have a clear sense of the validity of their review or ethics of their process.
• Sometimes KLAS is helpful for decision makers who know nothing about the vendor/product landscape. Otherwise, I never use it.
• I rarely if ever use it.

Generally Positive Comments

• We use KLAS as a data point in selecting a new vendor, but it is not the primary driver unless there are a significant number of negative comments or scores. We also monitor our current vendors to ensure they are keeping up with the market.
• I am using KLAS with a grain of salt and not as a gospel. For lack of a better reference frame, we all go to it, but I would not make decisions on KLAS alone. It is pretty much like the board certification for physicians: we all know that it may not reflect the best quality in a physician, but we all look it up and diligently go and take it to stay current.
• I view KLAS as just being one gauge on a dashboard when evaluating vendors.  For new vendor selections, KLAS is used to populate the initial list of potential vendors. Through the selection process, their rankings are used as a single data point, primarily as a reflection of market penetration, customer service, and overall satisfaction. I have to admit that I rarely refer to KLAS for vendor products we’ve implemented unless we’re experiencing issues or entertaining a product switch.
• I have used KLAS as a data point when evaluating vendors. For me, it represents a general standing in the marketplace and the comments are valuable in identifying areas to question.
• I view KLAS as a consolidated reference check. I provide feedback to KLAS on products and services once or twice a year and I know that how I replay can vary depending upon the most recent encounter with the vendor in question. As with any reference check, you get a good picture of how one or many are currently viewing the company. KLAS will never be the final word, but is a good place to go to get a consolidated view of how customers are feeling about the vendor.
• We use the KLAS scores as a starting place. We also use them as a resource to understand what other hospitals are doing. Adam Gale and his team are great about answering questions. They obviously have a great network of contacts and can often point us to other organizations who have addressed similar challenges.
• When private physician practices contact me for advice on EMR vendors that they are reviewing, I share with them the publically available KLAS reports as well as other industry reports on EMR metrics. I also use these reports to see if there is correlation between what is being reported and what is said in private and on HIStalk about the vendors.
• We incorporate the results as part of our customer communication and status updates. Specifically, we ask the leaders of our IS teams over each area (e.g., surgical services) to routinely incorporate market feedback from KLAS during their standing customer meetings. This is typically only done twice per year, not at each monthly discussion. It also helps us confirm/deny trends that we may or may not be seeing locally at our organization.
• I’ve used KLAS to identify competing products in a space if we are looking to meet a need. We’ve referenced some of the reports when going through vendor selection, but it has not been the deciding factor. I’ve also found the reports to be an encouragement that we’re in the same boat as others.
• I routinely review KLAS reports on all current vendors and ones we are looking at. It’s helpful to get updated information. Because I participate in KLAS reviews, I am able to get detailed reports related to vendors and trends. I’m usually looking for details on satisfaction with implementation and ongoing support. Love their question: would you buy from this vendor again?
• I review KLAS findings and typically drill down into the individual comments from other users to find information or concerns that I use with the vendors in order to get more specific information. For example, if a number of users complain about some aspect, then I may spend more time than I might otherwise have done drilling the vendor about that aspect. I can also occasionally find out what the vendor has problems with, and if I’m convinced it won’t be a problem for us (and that we want to go forward with them), I can occasionally use that to negotiate a better deal.
• I use KLAS primarily in the selection process for software and services and in that regard I find them very valuable, especially the user comments both pro and con. They give me some good direction in term of things I make sure I follow up on in the selection process. Recently they have also created some additional functionality around the creation of affinity group and other functional that gives me a platform to share directly with other organizations who have similar products or are similar to me in structure (academic, for example) that I have found some good utility in.
• (from a vendor employee) As a vendor, we do yearly, in-depth, anonymous, customer surveys to see how we truly stand in all areas of our solution, service, and support. That said, KLAS is incredibly helpful for us to get even further information on our performance. I find KLAS gets better executive level feedback than we get on our own (our surveys usually get more responses from managers/directors/end-users). It’s a great way for vendors to see objectively where they’re doing well and where they might have opportunities for improvement. I always tell folks, I love hearing all the great stuff about our company and solution but I’d much rather hear the “tough” stuff as that’s the gold that helps you become better and better.
• I use the KLAS reports to come up with a short list of vendors before the application/service search. The reports provide information that I use to educate my customers as to what is available, what others use in similar markets (e.g. practice EMR pool is different for 1-6 providers as compared to a practice of over 100 providers), as well as what applications others are moving from (always good to show there are no perfect vendors). I do peruse the vendor alerts as they come in but to this point I’ve not seen anything that was news to me.
• I use KLAS for independent ambulatory physicians who are looking for a system — it is excellent for them and they often do not know it exists. I also use it to go to battle when an operations person wants to buy a niche vendor system that I don’t want. (of course that only works if the KLAS scores are bad). Occasionally use it for our own purchases that I am trying to investigate, but unfortunately many of the systems we are looking to buy are not rated in KLAS (population health, analytics etc.)
• Used as one of the tools as part of vendor and system selection or standardization efforts. Also use Gartner info such as magic quadrant and we now ask IT vendors to register on VendorMate and pull reports on financial and sanction info from that resource and use Gartner for contract negotiation market analysis.
• I use KLAS infrequently, but it has served as a way to educate and inform our leadership about specific vendor offerings and their comparative value to the market. 
• KLAS scores and reports are critically important to me in my decision making process. They are my single most influential source of external advice and insight, followed by The Advisory Board and Gartner. KLAS’s integrity is unshakeable and their influence on the industry is invaluable.
• I review KLAS to identify top vendors meriting consideration and to yield additional insights into strengths and weaknesses when selecting vendors.
• Flawed, but extremely valuable given there’s no better alternatives in many cases. We used it a year ago to help determine whether we should go with a particular vendor on the outpatient side (we didn’t as their product was rated in the bottom of the rankings). The one area where KLAS is lacking is in specialty-specific EMR evaluations, as the niche products that are great don’t show up on the KLAS radar because of lower volumes. 
• I participate in KLAS surveys because the lady who calls used to work for me and I like her style and that of the company. I find the reports insightful and they help confirm our assessments and sometimes point out weaknesses. I am aware of some of the criticisms of KLAS and certainly recognize their limitations. It is also helpful in working with the senior team, who may see only the glitz. It helps when I show our own vendor’s ratings, with which they usually agree, as a means to establish a level of credibility in KLAS reports.
• I don’t have real decision-making power (e.g., authority, monetary control) over HIT purchases. However, as a physician end-user and member of our institutional EHR committees, I have used the KLAS reports as a "reality check" when my personal impression of a particular product is dramatically different from the party line that’s being perpetuated by our hospital IT group and C-suite. They say "This software’s perfectly reasonable, but the doctors are being resistant." It’s nice to be able to say, "I don’t think it’s just our doctors who view this software as having problems…." I would say that the KLAS reports are helpful in encouraging greater honesty and reality checking when too many folks are drinking a LOT of Kool-Aid.
• Use it on a limited basis for specialty systems and needs. Good reference point to check and confirm which vendors we should consider for a selection

Key Themes

• KLAS uses questionable and non-transparent methodology.
• KLAS is far from perfect, but has little competition.
• The negative comments and scores are more meaningful than the positive ones.
• It’s good for a quick check on what customers think.
• KLAS reports can help determine if a trend you’re seeing locally is broad.
• It’s a good starting point for researching a vendor or product type, but is not the deciding factor.
• New service to allow members to contact each other is useful.
• Use KLAS reports to identify available products of a particular type.
• Review the scores of IT-recommended systems to make sure they are being considered on merit and not IT department convenience.
• Use the reports to educate and influence users involved in selection.
• Show negative reports to users who are convinced that they want a particular system or to remind users that all systems have negatives and that implementing them is hard work.