I read with interest the recent alternative certification proposal from John Halamka.
I have a couple of vendor friends who work on the certification process for their respective organizations. They both describe the process as cumbersome and tedious. One of them is a nurse and says she detests the entire process since it forces adherence to rigid scripts rather than testing the actual workflows users are going to need. Those of us who have spent a bit of time implementing systems know there is a significant difference between vendor QA testing (where vendors see if the code that was produced meets the build specifications) and true user acceptance testing, where we see if the code that was produced actually meets the needs of those using the system.
Whenever I assist with running user testing events, I make sure we test features and functionality using a dual approach. Some users will be given turn-by-turn test scripts that target a new workflow component in the context of the larger existing workflow, to ensure that the new pieces don’t adversely impact any other parts of the workflow. We all know about releases that fix one thing and break another, and this seems to be the best way for many clients to catch those kinds of issues.
Another group of users will be given test scripts that are a bit more nonspecific, such as, “Prescribe these three medications, then schedule an appointment for an office visit and send a referral for a mammogram through the portal.” This approach allows us to test new features against the way users actually use the system rather than against a rigid test script.
Users are generally creative. If there’s a work-around to be found or an alternate way to do something, they’ll unearth it. Sometimes those workflows are legitimate – the vendor offers three or four different ways to do something. However, some work-arounds may take advantage of unintended functionality or existing defects, so that that when those seemingly-unrelated defects are fixed, it causes issues with other workflows. You’re generally not going to find those with rigid test scripts since you may not have any way of knowing how creative your users have gotten or what workflows they have come up with.
Of course, testing those kinds of scenarios is far beyond certification, and with as tedious as certification already is, I’m certainly not advocating expanding it. It’s just a shame though that vendors are spending time certifying their products against criteria that have little impact on the actual use of their product.
At the same time, we seem to be lacking in actual usability testing. Although vendors are being pushed to include user-centric design principles in their processes, the outcomes still vary widely. The recent dust-up with Athenahealth’s Streamlined upgrade seems to illustrate this. Judging from the comments I’ve seen and heard, it feels like there may not have been enough user acceptance testing to identify workflow problems that are causing significant issues for a good number of their clients.
Although the comments should be taken with a grain of salt (since it’s difficult to know whether clients attended training, performed testing, whether they were following best-practice workflows previously, etc.) there is always a kernel of truth to be found. I’ve been on the receiving end of enough poorly-conceived or poorly-executed vendor “enhancements” to know that they seem to make it out the door more often than they should.
Sometimes they are the product of good ideas. but the technology doesn’t really make them executable. Sometimes they are enhancements that were created for a single client as a result of a contractual obligation even though they have zero utility for the rest of the vendor’s customer base. Other times they are enhancements that were created for sales purposes, to allow for a glitzy demo that looks good yet doesn’t meet the needs of actual physicians or clinical users. Not only are they unhelpful, but a couple I’ve seen recently are downright insulting to the good sense of the average doc.
In his comments, Dr. Halamka discusses how certification has negatively impacted the industry: “Overly zealous regulatory ambition resulted in a Rule that has basically stopped industry innovation for 24-36 months.” Clients who have waited patiently for their vendors to implement basic usability enhancements know exactly what he’s talking about. Rather than improving the user experience, scarce development dollars were spent meeting the letter of the law for requirements that may never be used. He closes with some profound thoughts that made my day:
If Brexit taught us anything, it’s that over regulation leads to a demand for relief.
Pythagoras’ Theorem has 24 words
Archimedes’ Principle has 67 words
The Ten Commandments has 179 words
The US Declaration of Independence has 1,300 words
The EU regulation on the sale of cabbages has 26,911 words.
As a comparison, the 2015 Certification Rule document has 166,733 words.
Good food for thought for the governmental bodies, agencies, payers, and others whose rules define how we deliver healthcare in the US.
What do you think about excessive rulemaking? Email me.
Email Dr. Jayne.