Black Book’s Vendor Report Methodology
Readers asked how Black Book performs its survey-driven health IT vendor reports, apparently surprised that Allscripts finished in the #1 spot for integrated EHR/PM/RCM vendors.
Doug Brown is president and CEO of Black Book Market Research, LLC. He has a long background in hospital administration and a master’s degree in hospital and healthcare administration. He provided quite a bit of information and the full detail behind this particular report, responding nearly instantly to my email. He says the company received a dozen calls in one day about this report, which is just one of 140 it publishes each year.
My questions and Doug’s answers (paraphrased for conciseness by me except when in quotes) are below.
How do you choose the people you survey?
The company sends survey invitations – usually during the big conference season – to those who have volunteered. That includes 90,000 past participants; 330,000 website signups; contact lists obtained from membership groups, journal subscribers, conference attendees; and for private physician practices, contact information from third-party lists. Participants are required to provide a verified company email address for validation.
Are vendors involved, either in providing a client list or publicizing the survey?
Never, Doug says, and he invites anyone to ask any highly-ranked vendors if they’ve ever been in contact with Black Book. Black Book discourages vendor and public relations company involvement and doesn’t communicate with them as surveys are underway (and doesn’t ask them for client lists). He also adds that plenty of vendors publicize their #1 rankings without even buying the detailed report, which he says is just fine.
Black Book can’t restrict vendors from suggesting that their clients complete surveys, but it discourages the practice.
Do you have a sample questionnaire?
The company provided its standard list of 18 KPIs for software or services, which have remain unchanged since they were developed in 2010 with help from academics with relevant software and services experience. It may explain a given item differently based on the audience, such as an infection control nurse vs. a business office manager.
In the 18 principles under “support and customer care,” it is stated that “External analysts, press/media and other clients reference this vendor as a services leader and top vendor correctly.” Does that mean customers provide a response, or that this element isn’t provided by customers?
“The content under the 18 key performance indicators is meant to only be a guide and are modified occasionally to suggest ways that that KPI can be interpreted. For instance, if the analysts or other clients are highly satisfied in terms of support and customer care, so may you. They are suggestive ways to consider the KPI theme – such as reliability or trust. Our goal was to find aspects of the client experience that a prospective buyer could not find in vendor RFP responses or get from tainted vendor-provided client reference calls. We aim to find the user level experience from a wide response pool perceptions, -not the input of a couple dozen financial decision makers or CIOs on advisory boards.”
Was additional information used for the report on integrated ambulatory systems?
“After we are in the audit stages, we often go back to the survey respondents with some additional questions on trends and strategies to give the vendor results some additional color. You will find that in the report before the vendor rankings (much is in the press release) and feel free to share that info.”
The survey responses are reviewed immediately by both internal and external auditors for completeness, accuracy, and respondent validity. Responses from at least 10 unique clients are required to be named in the top 10. Sample sizes that fall below required limits are asterisked.
Overall vendor rank is based on the mean score of the 18 criteria. Each company’s rank in each of the 18 criteria is provided as well.
Some categories had interesting responses of the “wonder what they were thinking here?” types. You’ll have to obtain the full report for details, but I’m flabbergasted that four companies that finished well in the “viability and competent financial management and leadership” category either replaced top executives or sold themselves recently; the top finisher in data security was the only company to have gone offline due to a ransomware attack; and Epic failed to crack the top 10 in surprising categories, finishing behind some questionable players.
However, these are the responses of customers, so their impressions and willingness to remain customers is what counts most.
Here’s a sample category result. I removed the vendor information since that’s in the report that Black Book sells (and that they sent me).
Note that this particular survey really didn’t address EHR functionality, just the practice management capability of EHR-integrated systems. Also, it does not appear that vendors selling multiple product lines (Allscripts would top this category, as well) have their individual products broken out, so mixing Practice Fusion with TouchWorks may not yield a sound product-specific result.
Another potentially weak point is one that KLAS struggles with – can a given respondent answer all the questions accurately, such as IT people scoring training or a nurse opining on security?
I’m interested in your opinions.
Carol Harris, the GAO representative testifying, seemed comfortable estimating hundreds of millions of dollars to get the remaining 94% of…