CIO Unplugged 12/15/09

The views and opinions expressed in this blog are mine personally, and are not necessarily representative of Texas Health Resources or its subsidiaries.

Business Continuity is not Disaster Recovery
By Ed Marx

Colorado 1997. After six tennis matches in two days, I had played my way to the semi-finals of a USTA clay court tournament. The morning of finals, I parked in the lot then went to retrieve my gear from the trunk. I always carried three near-identical racquets to a tournament. If a string busted or the environment necessitated an adjustment, I’d simply grab another and keep playing. A great disaster recovery plan.

This time, however, the trunk wouldn’t unlock. I even tried unsuccessfully to take apart the backseat, which was not a disaster I’d planned for. The match started in 20 minutes, and I had no racket, nor had I warmed up. At least I had my wallet. I found the pro shop and borrowed a couple of demo racquets similar in weight and feel to my own, and then rushed to the court.

Texas 2009. I received a call from work delivering an automated message stating that a disaster had occurred. I immediately joined the virtual command center. Our corporate offices had been shut down and evacuated. This emergency impacted corporate and all our health system facilities within a five-mile radius, which included a hospital and our central business office. Oblivious to the fact our sixth hospital in seven months was Going Live on EHR and a revenue cycle management system, the crisis persisted.

Since corporate was located within five miles of the Dallas Cowboys Stadium, Six Flags over Texas and Rangers Ballpark in Arlington, planning for the worst-case scenario was a must. This drill evaluated how well the ITS division could respond. After significant annual trialing, we had the disaster recovery piece down pat, but we’d never tested our business continuity.

Confidence in our enterprise business continuity, however, could only come after we were prepared. We could never benefit our organization in a disaster if we were personally unable to operate in challenging circumstances.

Here are the lessons we learned:

Leadership

· Pressure reveals character. One whom I thought was a great leader had a complete meltdown. Conversely, one of our quiet leaders surprised the heck out of me. As the one most affected by the drill, she executed brilliantly

· Leaders should never leave their laptops at the workplace

· The borderless office was brilliant in hindsight

Technology

· Ensure you have enough licenses to handle increase in remote workers

· Ensure that all workers have access to systems from home (PC or laptop)

· All departments should incorporate use of remote technology and collaborative tools in daily practice

Communications

· Standardize calling trees and routinely review accuracy

· Call notification system should be branded with a familiar screen ID name so people answer the phone

· Call notifications should incorporate a minimum of 3 touch points per employee (cell; text & call, home phone, work email, home email, etc)

· Call notifications should have the ability to reach successive layers of leadership in the event primary responsible parties are non responsive

· Call plans should be backed up to flash drives and be kept with you always

· Established processes with corporate business continuity leadership to ensure coordination

· Include contractors in call notification processes

· Have multiple options for communications (traditional and 2.0) in the event your primary tools are unavailable

· Given the dependence upon technology in healthcare, set the expectation that knowledge workers are essentially on-call 24x7x365

Logistics

· Develop and routinely review coordination plan with hospitals space availability to house displaced workers

· Code worker badges to allow entrance into all hospitals

· Ensure all workers are comfortable with the remote technology both for traditional and nontraditional applications

· Purchase laptops for all IT workers. This is not the 90s!

Operations

· Ensure the Service Desk in particular is comfortable with business continuity

· Groups less prone to borderless offices tend to be the most unprepared for remote work (Service Desk, Field Services)

· Leaders should conduct more frequent leader-only drills to ensure they can run the organization remotely

· Drills should be conducted quarterly at a minimum so that everyone is mentally/physically prepared for the real thing

· Develop Business Continuity portal with step-by-step instructions on execution

· Seeing as disaster does not discriminate, do not allow exceptions for participation

Other

· Future drills should extend from 3 days to 30 days

· Make sure executives and hospital leaders are aware of the need for IT to conduct business continuity exercises, which may affect operations

· Include executives and hospital leaders in planning and coordination efforts

I was pleased with our first-ever business continuity drill performance, a significant learning experience. As healthcare IT workers, we had a massive responsibility and an obligation to our customers and patients; the show had to go on. I believe the drill better prepared us for the real thing.

Finally, no employee evaluation, leadership assessment, or 360-degree feedback analysis exists that can give as keen of insight as observing people under pressure. Some will search out the spare racquet and win the tournament while others will crumble under the pressure and double fault.

Determine who’s got talent before the next tournament.

Ed Marx is senior vice president and CIO at Texas Health Resources in Dallas-Fort Worth, TX. Ed encourages your interaction through this blog. (Use the “add a comment” function at the bottom of each post.) You can also connect with him directly through his profile pages on social networking sites LinkedIn and Facebook, and you can follow him via Twitter – User Name “marxists.”