I want to add the following. I'm an IT professional. In the course of my professional duties, I'm expected to…
Curbside Consult with Dr. Jayne 7/11/16
I read with interest the recent alternative certification proposal from John Halamka.
I have a couple of vendor friends who work on the certification process for their respective organizations. They both describe the process as cumbersome and tedious. One of them is a nurse and says she detests the entire process since it forces adherence to rigid scripts rather than testing the actual workflows users are going to need. Those of us who have spent a bit of time implementing systems know there is a significant difference between vendor QA testing (where vendors see if the code that was produced meets the build specifications) and true user acceptance testing, where we see if the code that was produced actually meets the needs of those using the system.
Whenever I assist with running user testing events, I make sure we test features and functionality using a dual approach. Some users will be given turn-by-turn test scripts that target a new workflow component in the context of the larger existing workflow, to ensure that the new pieces don’t adversely impact any other parts of the workflow. We all know about releases that fix one thing and break another, and this seems to be the best way for many clients to catch those kinds of issues.
Another group of users will be given test scripts that are a bit more nonspecific, such as, “Prescribe these three medications, then schedule an appointment for an office visit and send a referral for a mammogram through the portal.” This approach allows us to test new features against the way users actually use the system rather than against a rigid test script.
Users are generally creative. If there’s a work-around to be found or an alternate way to do something, they’ll unearth it. Sometimes those workflows are legitimate – the vendor offers three or four different ways to do something. However, some work-arounds may take advantage of unintended functionality or existing defects, so that that when those seemingly-unrelated defects are fixed, it causes issues with other workflows. You’re generally not going to find those with rigid test scripts since you may not have any way of knowing how creative your users have gotten or what workflows they have come up with.
Of course, testing those kinds of scenarios is far beyond certification, and with as tedious as certification already is, I’m certainly not advocating expanding it. It’s just a shame though that vendors are spending time certifying their products against criteria that have little impact on the actual use of their product.
At the same time, we seem to be lacking in actual usability testing. Although vendors are being pushed to include user-centric design principles in their processes, the outcomes still vary widely. The recent dust-up with Athenahealth’s Streamlined upgrade seems to illustrate this. Judging from the comments I’ve seen and heard, it feels like there may not have been enough user acceptance testing to identify workflow problems that are causing significant issues for a good number of their clients.
Although the comments should be taken with a grain of salt (since it’s difficult to know whether clients attended training, performed testing, whether they were following best-practice workflows previously, etc.) there is always a kernel of truth to be found. I’ve been on the receiving end of enough poorly-conceived or poorly-executed vendor “enhancements” to know that they seem to make it out the door more often than they should.
Sometimes they are the product of good ideas. but the technology doesn’t really make them executable. Sometimes they are enhancements that were created for a single client as a result of a contractual obligation even though they have zero utility for the rest of the vendor’s customer base. Other times they are enhancements that were created for sales purposes, to allow for a glitzy demo that looks good yet doesn’t meet the needs of actual physicians or clinical users. Not only are they unhelpful, but a couple I’ve seen recently are downright insulting to the good sense of the average doc.
In his comments, Dr. Halamka discusses how certification has negatively impacted the industry: “Overly zealous regulatory ambition resulted in a Rule that has basically stopped industry innovation for 24-36 months.” Clients who have waited patiently for their vendors to implement basic usability enhancements know exactly what he’s talking about. Rather than improving the user experience, scarce development dollars were spent meeting the letter of the law for requirements that may never be used. He closes with some profound thoughts that made my day:
If Brexit taught us anything, it’s that over regulation leads to a demand for relief.
Pythagoras’ Theorem has 24 words
Archimedes’ Principle has 67 words
The Ten Commandments has 179 words
The US Declaration of Independence has 1,300 words
The EU regulation on the sale of cabbages has 26,911 words.
As a comparison, the 2015 Certification Rule document has 166,733 words.
Good food for thought for the governmental bodies, agencies, payers, and others whose rules define how we deliver healthcare in the US.
What do you think about excessive rulemaking? Email me.
Email Dr. Jayne.
“At the same time, we seem to be lacking in actual usability testing. Although vendors are being pushed to include user-centric design principles in their processes, the outcomes still vary widely.”
Usability arises from incorporation of user-centered design (UCD) principles as well as formative and summative testing in the development and deployment of systems.
(NISTIR 7741) NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records “provides NIST guidance for those developing electronic health record (EHR) applications that need to know more about processes of user-centered design (UCD). An established UCD process ensures that designed EHRs are efficient, effective, and satisfying to the user. Following the guidance in this document will greatly increase the likelihood of achieving the goal of building usable user interfaces and a better user experience. One of the main purposes of this guide is to provide practical guidance on methods relating to UCD and usability testing. The intended audiences of this document are those with a role in determining the features and functions contained in the EHR and how those are represented in the user interface.”
http://www.nist.gov/manuscript-publication-search.cfm?pub_id=907313
> Pythagoras’ Theorem has 24 words
> Archimedes’ Principle has 67 words
> The Ten Commandments has 179 words
> The US Declaration of Independence has 1,300 words
> The EU regulation on the sale of cabbages has 26,911 words.
> As a comparison, the 2015 Certification Rule document has 166,733 words.
So simpler “laws” and concepts take fewer words to explain? Real earthquake from Dr. Halamka there. I appreciate that we are looking at the face of excessive rules, and I don’t dispute that this is a problem. But there’s no comparison between the Pythagorean Theorem and the Certification Rule. Nor between almost any two of the things in this list. This kind of hyperbole gets us nowhere. Less is not always more, but then, neither is more always more.
While I appreciate the suggestion that Brexit wasn’t pure xenophobia, but (significantly) a desire for reduced regulation, the Cabbage Regulation claim is (unfortunately) a hoax. See http://www.snopes.com/language/document/cabbage.asp.
So many of your observations in this post are spot on. I work for a vendor and some of our regulatory projects meet the letter of the regulation but are nearly unusable. Meaningful Use Patient Education and Infobutton is a terrific example; just because it’s technically possible for a system to copy Education information from an external website and log that action doesn’t mean it’s easy, user friendly, or actually in practice.
Software vendors should be accountable first to their end users. If the software doesn’t work for them, it’s not going to meet any outside objectives or mandates. The best way to accomplish that is by iterative agile design with end user testing before launch. Regulators can’t mandate this, it has to evolve as a business practice. This is a lesson our industry has been slow to learn, but once we do, we can make a big difference in the daily workflow of medical professionals.
No surprise here…when did the gvt ever create a regulation that made things simpler??
And here’s the real irony. The HiTech Act was based on work that was done by none other than HIMSS and it’s vendor cohorts. Remember CCHIT?? Who created that and told the gvt they could use it as the basis for designating certified systems?
And now vendors complain…as I said many times on this blog when you choose to sleep with a devil, you’ll always end up in H*LL.
I appreciate the clinician perspective on usability testing and the entire UAT workflow. It dovetails nicely with my experience.
While I have always advocated for usability from the technology side of the house, it has (too often) been a struggle. Oftentimes we have clients who have no technology skills or interest. Or they aren’t engaged with the software lifecycle and ‘just assume’ that IT will do the right thing. And IT management itself sometimes isn’t helpful because they ‘want the product out the door’ or some other image-based goal.
Quality and usability take time. It takes dedication and focus. And IT needs a partner to steer us because, no matter how experienced we are, we aren’t performing your job and we never will. Just as you won’t do our job; it’s not a reasonable expectation.