In the last several weeks, tornadoes and other severe storms have ripped through various parts of the country. Based on a frantic phone call I received from a medical school colleague, this seems to be a good opportunity for a physician-friendly discussion of disaster preparedness for healthcare information technology. For those of you who are serious IT professionals, this may be boring, but on the other hand it may be a good conversation starter to e-mail (or even print if you have to) for the physicians in your lives.
Downtime and Disaster Recovery 101
The most important part of successfully dealing with an outage of your electronic health record is to have a plan. Most practices need both short-term and long-term plans, whether you’re in a well-known hurricane zone or tornado alley or not. Lots of things can happen: floods, fires, and earthquakes. No one is exempt and everyone needs a plan.
Downtime usually refers to a time when the system is unavailable, whether planned or unplanned. Downtimes can happen for a variety of reasons. Unplanned downtimes may include a local power outage, loss of Internet connectivity, or other nonspecific system issues that keep physicians from fully using the EHR. They may be limited — perhaps it’s just an outage of e-prescribing or faxing — or may affect the system across the board.
Limited downtime events often have simple workarounds. For example, if e-prescribing or faxing is down, one can always print prescriptions or documents, call medication orders to the pharmacy, or worst-case scenario (ugh) use a paper prescription pad and a pen. Loss of Internet connectivity can be overcome by using a cellular / wireless Internet card, provided the practice has planned ahead and such cards are available for use. If the local wireless network in the practice is out of commission, users may be able to plug in, assuming there are ports available.
For unplanned downtimes, unless they have in-house IT support 24×7, practices should ensure key personnel have checklists for troubleshooting issues and phone lists for Internet service providers, vendor help desks, etc. Make sure multiple people in the practice know how and where the information is stored — don’t count on a single employee to be the point of knowledge. Murphy’s Law dictates that if something goes wrong, it will go wrong when that employee is unavailable.
Planned downtimes are usually limited downtimes. This may include hardware upgrades, software upgrades, weekly or monthly maintenance, etc. When planning a downtime, physicians need to discuss their willingness to work without full access to the EHR. Many physicians may be willing to print summaries for patients who may be scheduled during an upgrade and ‘wing it’ for others. For some, being without data is unacceptable, and the office must be closed.
Careful planning can keep physicians from having to make this decision. Many vendors offer solutions where a copy of the database can be saved to a local computer and accessed in a read-only manner during an upgrade. There are several third-party solutions to this problem, and if you are interested in this for your practice, allow some time (often a few months) to make sure this is in place before a planned downtime.
Disaster recovery usually refers to a situation where something very, very bad has happened. This can include physical destruction of the practice, its servers, and its equipment due to a natural disaster. If the IT infrastructure is physically destroyed, it may be weeks before the practice can be up and running. Disasters can also occur due to poor planning, as my friend learned.
Practices need a plan to create backup copies of the data in the event of a disaster. If you use a Web-based or hosted EHR, often your vendor takes care of backups for you. However, you need to understand the interval at which backups are done. Daily, weekly, monthly? To determine how frequently you need to do a backup, ask yourself: how much data are you willing to lose? For a busy practice, backups should be done daily and practices should consider other strategies to continuously back up data throughout the day (but that’s beyond Disaster Recovery 101, so I’ll save the discussion of transaction log shipping vs. database mirroring for another day).
Backups should not be stored in the office. Think it through: if your office catches on fire and the backup copy is at the office, that’s not a great idea. Backups need to be stored securely under appropriate climate conditions — be mindful of temperature, humidity, etc. There is one important thing about backups that doesn’t cross most physician minds: the need to test the backup to make sure it works. Your IT professionals can do this by taking the backup copy of the database and restoring it to a test system, then checking it to make sure data is current and comprehensive.
Unfortunately, the solo physician who called me this morning learned this the hard way. When the power went out and the battery backup failed, the database was impacted. Her vendor recommended that they restore the database from the most recent backup. When this was attempted, the backup contained less than half the data they expected it to. Not a great situation. Although she was fortunate that the EF-4 tornado didn’t touch her building, it’s going to be a challenge to recover from the loss of so much data.
So physicians, heed this cautionary tale. Take a moment to discuss your downtime and disaster recovery strategies with your IT support staff, whether you work in a solo practice or for a large health system. Don’t be afraid of stepping on the IT team’s toes — many are proud of the downtime strategies they’ve created and will be happy to talk about them. If there is no written plan, make it a point to create and document the processes you need to practice should the system be unavailable. Make sure key staff have copies of the plan, and practice it. Use regular maintenance windows as an opportunity to practice what you would do if an unplanned outage occurred.
Preparing for system outages should be a regular part of the life of the practice, no different than fire drills, tornado drills, or the like. The odds of something bad happening may be slim, but if you’re in disaster’s crosshairs, you’ll be glad you took the time to prepare for the worst and to protect your patients and your practice.