Home » News » Currently Reading:

Black Book’s Vendor Report Methodology

November 15, 2018 News 7 Comments


Readers asked how Black Book performs its survey-driven health IT vendor reports, apparently surprised that Allscripts finished in the #1 spot for integrated EHR/PM/RCM vendors.

Doug Brown is president and CEO of Black Book Market Research, LLC. He has a long background in hospital administration and a master’s degree in hospital and healthcare administration. He provided quite a bit of information and the full detail behind this particular report, responding nearly instantly to my email. He says the company received a dozen calls in one day about this report, which is just one of 140 it publishes each year.

My questions and Doug’s answers (paraphrased for conciseness by me except when in quotes) are below.

How do you choose the people you survey?

The company sends survey invitations – usually during the big conference season – to those who have volunteered. That includes 90,000 past participants; 330,000 website signups; contact lists obtained from membership groups, journal subscribers, conference attendees; and for private physician practices, contact information from third-party lists. Participants are required to provide a verified company email address for validation.

Are vendors involved, either in providing a client list or publicizing the survey?

Never, Doug says, and he invites anyone to ask any highly-ranked vendors if they’ve ever been in contact with Black Book. Black Book discourages vendor and public relations company involvement and doesn’t communicate with them as surveys are underway (and doesn’t ask them for client lists). He also adds that plenty of vendors publicize their #1 rankings without even buying the detailed report, which he says is just fine.

Black Book can’t restrict vendors from suggesting that their clients complete surveys, but it discourages the practice.

Do you have a sample questionnaire?

The company provided its standard list of 18 KPIs for software or services, which have remain unchanged since they were developed in 2010 with help from academics with relevant software and services experience. It may explain a given item differently based on the audience, such as an infection control nurse vs. a business office manager.

In the 18 principles under “support and customer care,” it is stated that “External analysts, press/media and other clients reference this vendor as a services leader and top vendor correctly.” Does that mean customers provide a response, or that this element isn’t provided by customers?

“The content under the 18 key performance indicators is meant to only be a guide and are modified occasionally to suggest ways that that KPI can be interpreted. For instance, if the analysts or other clients are highly satisfied in terms of support and customer care, so may you. They are suggestive ways to consider the KPI theme – such as reliability or trust. Our goal was to find aspects of the client experience that a prospective buyer could not find in vendor RFP responses or get from tainted vendor-provided client reference calls. We aim to find the user level experience from a wide response pool perceptions, -not the input of a couple dozen financial decision makers or CIOs on advisory boards.”

Was additional information used for the report on integrated ambulatory systems?

“After we are in the audit stages, we often go back to the survey respondents with some additional questions on trends and strategies to give the vendor results some additional color. You will find that in the report before the vendor rankings (much is in the press release) and feel free to share that info.”


The survey responses are reviewed immediately by both internal and external auditors for completeness, accuracy, and respondent validity. Responses from at least 10 unique clients are required to be named in the top 10. Sample sizes that fall below required limits are asterisked.

Overall vendor rank is based on the mean score of the 18 criteria. Each company’s rank in each of the 18 criteria is provided as well.

Some categories had interesting responses of the “wonder what they were thinking here?” types. You’ll have to obtain the full report for details, but I’m flabbergasted that four companies that finished well in the “viability and competent financial management and leadership” category either replaced top executives or sold themselves recently; the top finisher in data security was the only company to have gone offline due to a ransomware attack; and Epic failed to crack the top 10 in surprising categories, finishing behind some questionable players.

However, these are the responses of customers, so their impressions and willingness to remain customers is what counts most.


Here’s a sample category result. I removed the vendor information since that’s in the report that Black Book sells (and that they sent me).

Note that this particular survey really didn’t address EHR functionality, just the practice management capability of EHR-integrated systems. Also, it does not appear that vendors selling multiple product lines (Allscripts would top this category, as well) have their individual products broken out, so mixing Practice Fusion with TouchWorks may not yield a sound product-specific result.

Another potentially weak point is one that KLAS struggles with – can a given respondent answer all the questions accurately, such as IT people scoring training or a nurse opining on security?

I’m interested in your opinions.


HIStalk Featured Sponsors


Currently there are "7 comments" on this Article:

  1. Mr. HISTalk

    Navin, Haffty & Associates was pleased to be named #1 for MEDITECH Implementation, Strategy and Support Solutions in their most recent report, Survey Period, Q1 – Q3 2018
    Prior to seeing this tweeted out by Black Book, I was not familiar with them. I can confirm that we were not asked to identify clients and only after the report was published, have we contacted them to learn more. We have never paid them for any services. Whether their methodology is good or not, I’ll leave to others to decide but can attest to their statement that vendors were not involved in generating responses. Feel free to post if you’d like or edit so it doesn’t come across as self-serving on my part. Just thought you should know

  2. I would like to see a Histalk survey posted that asks readers if they now perceive Black Book with more or less credibility due to this report. Having first hand use of Allscripts I can say this wraps up my desire to read any of their reports.

  3. I’ve tried for months to have any contact with Black Book to better understand how they determine which services we offer or in which reports we might be included – and then to what extent we can use their results in any marketing. They have been continually unresponsive. I guess that looks good to their audiences, but it’s frustrating if you just want to be sure they are covering you in all the areas you are active in the market. I think there is a middle ground somewhere.

  4. I just get red flags all over the place from this:
    * how does 2,194 round to 3,000?
    * why use two decimal places on KPIs when the n is about 15 responses on average?
    * tried their app. It crashed.
    * The idea that they discourage gaming the (overtly lax) system feels like lip service.
    * how do they keep out vendors with provider-domain email accounts?
    * methodology description… 10 unique clients one place and 5 unique clients another?
    * no client-size segmentation or respondent-type segmentation (1-doc office clerk perspective = IDN CEO perspective?)
    * does the report itself have n’s per vendor? Do they roughly map to market share?

  5. Thank you on the follow up and explaining what Black book says the process is.

    From my own personal experiences being around and using Allscripts products again NONE of their products are remotely close to being seamlessly, fully integrated.

    The product lines of ProEHR, ProPM and Touch Works use the same db vendor but they are each their own autonomous environments leaning/collapsing heavily on a plethora of questionably architected and mostly crashing services.

    Practice Fusion I can not comment on but another commentator here already pushed the question of What size are the participants in this survey.? Since a valid email is all you needed to create a Practice Fusion account then the process of sending surveys to random email address my be the reason behind Allscripts Ranking. Having a survey which may be manipulated by many minions greatly diminish the value of a survey intended to be industry encompassing.

    The question on how Black Book handles “hosted/vendor domain provided” client email address needs to be outlined.

    If the vendor being referenced in ” the top finisher in data security was the only company to have gone offline due to a ransomware attack;” is Allscripts then YES Black Book needs to go back to their academics and reinvent themselves.

    With a dwindling client base, very little new sales in US or abroad it is hard to believe anything about this survey and the process used.

    Thank you for the good work.

    • Hi, how are you positive that their client base is dwindling? I’m curious where you’re getting this information from. It appears to be so from the most recent 10-Q, but I’m wondering where you’re seeing this specifically. Thanks for your help.

Text Ads


  1. I share your concerns about a lot of similar projects, but I think this one's an outcomes comparison thing, not…

  2. "Ascension Via Christi sent several ICU nurses home after they raised concerns about inadequate staffing." Nice. That'll show em.

  3. Thank you for sharing the full HIStory file. What a great tribute to Mr. Ciotti. “Those who cannot remember the…

  4. "Epic will roll out a “patients like this one” type tool to its users in Indiana this summer. It will…

  5. I don't see Cerner winning employee satisfaction surveys. I can't see VHA pivoting to Epic. It's going to be some…

Founding Sponsors


Platinum Sponsors








































Gold Sponsors









RSS Webinars

  • An error has occurred, which probably means the feed is down. Try again later.