Time Capsule: Vendors: Develop an "Antivirus" Program to Warn of Your Software’s Bugs

I wrote weekly editorials for a boutique industry newsletter for several years, anxious for both audience and income. I learned a lot about coming up with ideas for the weekly grind, trying to be simultaneously opinionated and entertaining in a few hundred words, and not sleeping much because I was working all the time. They’re fun to read as a look back at what was important then (and often still important now).

I wrote this piece in June 2007.

Vendors: Develop an "Antivirus" Program to Warn of Your Software’s Bugs
By Mr. HIStalk

CIOs and vendors spend lots of money and time addressing system redundancy. Thank goodness. When clinical systems go down, patient risk goes up (or so you hope, anyway, since that’s the ultimate validation of that system’s influence on patient outcomes.)

We can agree that downtime is bad. What vendors haven’t fully acknowledged, however, is that systems can be up and running, yet still endangering patients because of internal logic or data errors. Known bugs, in other words, that cause errors that are subtler and, for that reason, potentially more dangerous.

I’ve worked in vendor support, so I’ve seen hundreds of examples:

• Medication doses, lab tests, or nursing actions are omitted from their respective printed or online schedules
• Interface problems allow time-critical information to be delayed or ignored
• Lack of sufficient storage space causes transactions to be lost
• Patient merging or discharge cancellation does something undesirable to visit information
• Technical issues cause background processing to fail, delaying reports or updates until a user finally notices.

I could go on, here’s my point. The vendor’s support people knew about these problems. They could get on a client’s system, query the database, and say, "Yep, we’ve seen this problem before." In most cases, a simple utility program could have detected the error condition proactively and warned someone immediately, allowing the problem to be resolved before patients were exposed to data-driven risk.

Vendors don’t like this idea. First, it admits that their software has bugs (which it always does.) Secondly, it also admits that even well-documented bugs can’t always be fixed immediately.

I don’t buy the idea that it’s the customer’s job to find and report problems. It’s never acceptable to endanger a patient with a known software defect, even if a fix is on the way. The obvious solution (temporary or otherwise) is to write a program to detect the problem, let the customer choose how often to automatically run it, and provide the appropriate alert when that problem is found.

Here’s a simple example. Suppose I have CPOE and pharmacy systems that should always be synchronized via complex interfacing or integration. That’s great, but what if something goes wrong? The unacceptable answer: let the clinician find the problem. Oops, the antibiotic order is active in CPOE, but expired in pharmacy. Customer support: "Thanks for telling us, but we already know about that problem, even though we can’t fix it. Continue to be vigilant. Can we close your case?"

This is not necessary. A program could easily have detected the problem. Programs are better than clinicians at comparing List A to List B. So, why are preoccupied clinicians expected to be the safety net for programmers?

None of the applications I’ve used provided these low- level diagnostics. Finding bugs was a user’s problem, even after those problems had been documented, acknowledged as bugs, and scheduled for an eventual fix.

This drives users through the roof. IT is proclaimed as unresponsive and the vendor is branded as incurably stupid.

My message to vendors: It’s your job to tell customers when your software has a problem, not vice versa. Ask your support reps for a list of known problem symptoms. Get your developers to write a diagnostic program that users can run on a predefined schedule, including their preference for alerts.

Think of it like your PC’s antivirus program. It has a core detection engine. Users determine how often it runs and what happens when it finds a problem. Automatic updates to its detection patterns let it find even newly discovered problems immediately.

Developing a problem detection engine isn’t an admission of failure. it’s a reflection of reality. Software always has bugs that leave detectable tracks in the customer’s database. Finding the occurrence of those clinical software bugs is good for everyone involved, most notably the patient.