Curbside Consult with Dr. Jayne 1/25/21

Many of my healthcare IT colleagues are deeply involved in their organizations’ COVID vaccine administration efforts. They’re involved in creating pre-registration and wait list systems, running analytics to determine who should be invited to receive a vaccine next, managing outreach efforts, then scheduling those patients. It’s a massive effort that, like many IT projects, can be subject to external disruptions.

One of our local health systems just went through a massive cancellation of vaccination appointments after they received fewer doses from the state than they anticipated. The sheer volume of appointments that had to be canceled and rescheduled created havoc. As their API-driven chatbot was reaching out to patients to offer them new appointments, it was creating temporary locks on the appointment slots that were being offered, which is standard when you’re doing scheduling outreach. However, the magnified consequence of trying to reschedule thousands of patients at once prevented the call center from being able to reschedule anyone else, including patient-facing healthcare providers who were needing to also reschedule after missing vaccine appointments while awaiting negative COVID tests.

The answer to the latter problem became decidedly low-tech, with the system standing up a temporary walk-in vaccine clinic to accommodate the healthcare providers with its remaining available doses. A Google Doc was used to keep track of the employees who were approved to come to the clinic and what time they planned to come, so that the vaccination team could coordinate with the call center to ensure that the correct number of doses were available real time. Since they weren’t running the public-facing vaccine clinic, they had a surplus of workers who could handle the manual scheduling, but the fact that the situation arose at all shows how much difficulty the US is having with the last mile of vaccine distribution.

With recent stories about vaccine spoilage due to temperature issues, those running the vaccination operations could learn from their IT colleagues. A Veterans Affairs hospital in Boston recently had a freezer failure from multiple contributing causes. First, a pipe burst leading to a water leak in the building, which led to the arrival of a cleaning crew who had to move a freezer to get to some standing water. The power cord for the freezer apparently wasn’t properly secured to the freezer, causing it to disconnect. Then the freezer’s alarm system didn’t function properly, which coupled with the lack of daily monitoring, led to the loss of 1,900 doses of vaccine.

The VA is still investigating why the alarm failed, but proper daily human monitoring could have saved the vaccines since the freezer was unplugged for several days before being discovered. Any small-practice primary care physician who has had to maintain thousands of dollars of vaccine inventory knows that even though you have thermometers with alarms, you still need to have someone check the logs daily and document the ranges. It’s shocking that a larger organization that is responsible for such a precious commodity didn’t have the right processes in place. However, based on some of the IT failures I’ve seen over the last several years, I’m not surprised.

Many healthcare organizations have complex automated backup systems and sophisticated disaster recovery systems that promise a rapid fail-over to sustain clinical operations. However, they may not test them often enough, and some organizations don’t test them at all. We’ve all heard horror stories of clients who went to restore from a backup, only to find that the backup contained no data or was corrupted in some way.

We’ve also encountered the unforeseen. Early in my career, a car that was involved in a police chase crashed into our hospital’s data center, which led to a small fire, which led to discharge of the fire suppression system and a complete shutdown of the building. There was a failure of the network switches that should have rerouted everyone to the disaster recovery site as well, which led to a multi-hour outage since no one could get in there to see if they could switch things manually since the building was now a crime scene. I’m sure “what if the building becomes a crime scene” was never in the minds of those who designed the downtime policies and systems, but you can bet it’s on the checklist for my consulting clients.

Organizations may also be missing physical safeguards that are needed for their systems to be effective, like the VA hospital’s freezer was missing a couple of screws that could have prevented the vaccine loss. I worked with a client not too long ago that thought they were creating nightly backups of their system. They were using removable hard drives as the media. An employee would come in every morning and disconnect the drive, place it in a manila envelope with the date, then pull the oldest backup drive and connect it to the system. They failed to lock the door to the data room consistently, however, resulting in the disappearance of the box full of envelopes and drives.

As I tell these stories, I feel a bit like a Monday-morning quarterback. However, except for the crime scene part, the preventive maneuvers for these situations are already well documented. HIPAA requires a Security Risk Assessment where covered entities must look at physical, administrative, and technical safeguards for protected health information. Participation in federal and state vaccine programs requires signing agreements on vaccine storage and accountability. Although there were technical failures in the situations above, the human error component is strong as well.

This story out of Boston isn’t the only vaccine loss story out there. Much larger losses were recently documented in Maine and Michigan. The COVID vaccine is such a scarce commodity. If I were in charge of an organization that was a vaccinator, you can bet that I would have daily touch points with the leaders involved to ensure accountability and that systems were in place to approach zero waste. Every dose that doesn’t go to someone who wants it is a tragedy in the making.

My parents and elderly relatives are scheduled for vaccines over the next two weeks. I’m crossing my fingers that they don’t get caught in one of these situations. Based on the horrors I see in my clinical role, I’ll be holding my breath to some degree until everyone in my family is fully vaccinated.

How does your organization approach disaster recovery planning? Do you have plans in place if you need to execute a massive rescheduling operation if vaccines are lost or delayed? Leave a comment or email me.

Email Dr. Jayne.