Home » Dr. Jayne » Currently Reading:

Curbside Consult with Dr. Jayne 1/25/21

January 25, 2021 Dr. Jayne 3 Comments

Many of my healthcare IT colleagues are deeply involved in their organizations’ COVID vaccine administration efforts. They’re involved in creating pre-registration and wait list systems, running analytics to determine who should be invited to receive a vaccine next, managing outreach efforts, then scheduling those patients. It’s a massive effort that, like many IT projects, can be subject to external disruptions.

One of our local health systems just went through a massive cancellation of vaccination appointments after they received fewer doses from the state than they anticipated. The sheer volume of appointments that had to be canceled and rescheduled created havoc. As their API-driven chatbot was reaching out to patients to offer them new appointments, it was creating temporary locks on the appointment slots that were being offered, which is standard when you’re doing scheduling outreach. However, the magnified consequence of trying to reschedule thousands of patients at once prevented the call center from being able to reschedule anyone else, including patient-facing healthcare providers who were needing to also reschedule after missing vaccine appointments while awaiting negative COVID tests.

The answer to the latter problem became decidedly low-tech, with the system standing up a temporary walk-in vaccine clinic to accommodate the healthcare providers with its remaining available doses. A Google Doc was used to keep track of the employees who were approved to come to the clinic and what time they planned to come, so that the vaccination team could coordinate with the call center to ensure that the correct number of doses were available real time. Since they weren’t running the public-facing vaccine clinic, they had a surplus of workers who could handle the manual scheduling, but the fact that the situation arose at all shows how much difficulty the US is having with the last mile of vaccine distribution.

With recent stories about vaccine spoilage due to temperature issues, those running the vaccination operations could learn from their IT colleagues. A Veterans Affairs hospital in Boston recently had a freezer failure from multiple contributing causes. First, a pipe burst leading to a water leak in the building, which led to the arrival of a cleaning crew who had to move a freezer to get to some standing water. The power cord for the freezer apparently wasn’t properly secured to the freezer, causing it to disconnect. Then the freezer’s alarm system didn’t function properly, which coupled with the lack of daily monitoring, led to the loss of 1,900 doses of vaccine.

The VA is still investigating why the alarm failed, but proper daily human monitoring could have saved the vaccines since the freezer was unplugged for several days before being discovered. Any small-practice primary care physician who has had to maintain thousands of dollars of vaccine inventory knows that even though you have thermometers with alarms, you still need to have someone check the logs daily and document the ranges. It’s shocking that a larger organization that is responsible for such a precious commodity didn’t have the right processes in place. However, based on some of the IT failures I’ve seen over the last several years, I’m not surprised.

Many healthcare organizations have complex automated backup systems and sophisticated disaster recovery systems that promise a rapid fail-over to sustain clinical operations. However, they may not test them often enough, and some organizations don’t test them at all. We’ve all heard horror stories of clients who went to restore from a backup, only to find that the backup contained no data or was corrupted in some way.

We’ve also encountered the unforeseen. Early in my career, a car that was involved in a police chase crashed into our hospital’s data center, which led to a small fire, which led to discharge of the fire suppression system and a complete shutdown of the building. There was a failure of the network switches that should have rerouted everyone to the disaster recovery site as well, which led to a multi-hour outage since no one could get in there to see if they could switch things manually since the building was now a crime scene. I’m sure “what if the building becomes a crime scene” was never in the minds of those who designed the downtime policies and systems, but you can bet it’s on the checklist for my consulting clients.

Organizations may also be missing physical safeguards that are needed for their systems to be effective, like the VA hospital’s freezer was missing a couple of screws that could have prevented the vaccine loss. I worked with a client not too long ago that thought they were creating nightly backups of their system. They were using removable hard drives as the media. An employee would come in every morning and disconnect the drive, place it in a manila envelope with the date, then pull the oldest backup drive and connect it to the system. They failed to lock the door to the data room consistently, however, resulting in the disappearance of the box full of envelopes and drives.

As I tell these stories, I feel a bit like a Monday-morning quarterback. However, except for the crime scene part, the preventive maneuvers for these situations are already well documented. HIPAA requires a Security Risk Assessment where covered entities must look at physical, administrative, and technical safeguards for protected health information. Participation in federal and state vaccine programs requires signing agreements on vaccine storage and accountability. Although there were technical failures in the situations above, the human error component is strong as well.

This story out of Boston isn’t the only vaccine loss story out there. Much larger losses were recently documented in Maine and Michigan. The COVID vaccine is such a scarce commodity. If I were in charge of an organization that was a vaccinator, you can bet that I would have daily touch points with the leaders involved to ensure accountability and that systems were in place to approach zero waste. Every dose that doesn’t go to someone who wants it is a tragedy in the making.

My parents and elderly relatives are scheduled for vaccines over the next two weeks. I’m crossing my fingers that they don’t get caught in one of these situations. Based on the horrors I see in my clinical role, I’ll be holding my breath to some degree until everyone in my family is fully vaccinated.

How does your organization approach disaster recovery planning? Do you have plans in place if you need to execute a massive rescheduling operation if vaccines are lost or delayed? Leave a comment or email me.

Email Dr. Jayne.

HIStalk Featured Sponsors


Currently there are "3 comments" on this Article:

  1. Building effective disaster recovery and highly reliable systems both require a high-availability mindset. I worked in a large organization that was doing a huge EHR rollout. Our IT organization didn’t have a great rack record of building and operating systems at a 99.9% availability. We didn’t do a good job with Root Cause Analysis. Our change control process wasn’t strict enough. There wasn’t good feedback into the development and platform engineering groups to prevent future problems. Our DR planning also suffered from blind spots but they hadn’t been exposed by a major disaster event in many years. It took a concerted effort led by IT leadership to get everyone involved to think differently about availability including DR. We brought in outside experts for training on high availability thinking. We used one of the definitive books in the field, by Karl Weick. We beefed up our RCA process and made sure the focus was on improvement, not punishing people for having made a human mistake. (A few people who willingly violated clear rules and caused outages were fired.). We turned the corner and kept up with the EHR rollout so it was successful. DR preparations became much more thoughtful. We are still vulnerable to the careless contractor with a backhoe who cuts fiber to a building, but with the right mindset you understand the risks of things like that and design accordingly.

  2. I have a completely different mindset regarding COVID vaccine supply shortfalls. Not from you Dr. Jayne, at least not that I can tell so far.

    Think about it this way: This is the Problem You Want To Have!

    Sure, in a perfect world, we’d have enough vaccine. This isn’t a perfect world though. And if we had excess vaccine, we’d be short of Physicians, or Nurses, or Pharmacists, or Public Health Offices. You are always going to be short of something!

    If you have to be short of something, you want to be short of vaccine. We can get more vaccine. I’m confident of that now. Now the converse: You are short of Physicians. How long does it take to train a Physician? How much money does it take to train a Nurse or Pharmacist? What are the hurdles you need to jump to open a new PH Office? It’s all difficult, costly, and there are year’s long lead times.

    According to the Milken Institute (via SciAm June 2020 issue, “Fast Track Drugs”), there were 133 experimental therapies as of April 2020. There were 49 in clinical trials. Holy Cow!

    We will be up to our eyeballs in vaccines and treatments very soon. I’m guessing, by Summer 2021. And all those Physicians, Nurses, Pharmacists, and Public Health Offices, will be waiting. They will scale up the vaccine rollout like crazy.

    No, Help is On The Way. The only thing missing is vaccine supply and that will become a veritable tsunami, very shortly. And there’s a whole system ready and waiting to spring into action.

    • Further to my forecast above.

      The Moderna and Pfizer vaccines have approval in both Canada and the U.S. AstraZeneca and Johnson & Johnston are on the verge of approval (I believe AZ has been approved in Canada, while J&J has been approved in the US. But I only follow these intermittently and I could be wrong).

      There is a Canadian vaccine that is tentatively slated for availability in 2022. Don’t ask me the name but it’s not from one of the major drug companies. Meanwhile, Sputnik is finally completing Stage 3 trials and looks pretty good. It’s even possible that Chinese vaccine (SinoVax?) could get approval here. Though I seem to recall questions about it’s efficacy.

      Availability is only going to get better over time.

      In just a few months, the big challenge will be Anti-Vaxxers and the Vaccine Hesitant.

Founding Sponsors


Platinum Sponsors




























































Gold Sponsors












Reader Comments

  • Brian Too: "...a physician who claims that vaccines cause people to become magnetic..." That physician meant "a magnetic persona...
  • State Board Skeptic: The state medical boards don’t really care about patient safety or good physician practice. What they do care about is...
  • DrM: State medical boards are now the self-protecting monopolies people fear when they talk about trades self-managing who ca...
  • Brian Too: I have occasionally been subject to nervous effluent, but never to nervous affluent!...
  • SJ: What a great write-up. This was helpful on many levels, especially to understand a clinicians point of view!! I always r...

Sponsor Quick Links