Home » Ed Marx » Currently Reading:

CIO Unplugged 1/13/11

January 13, 2011 Ed Marx 20 Comments

The views and opinions expressed in this blog are mine personally and are not necessarily representative of current or former employers.

Crisis Reveals Leadership

I finished my first week as CIO exhilarated. I slipped out early and headed for a haircut (I had hair back then). The grating buzz of the “emergency broadcast system” disrupted WTAM’s sports update. A power outage that began in the Northeast had hit Ohio. This was not a test.

Out the window, I watched traffic come to a halt. Electricity stopped, rendering signals colorless. I called my family and staff, but cell networks were overwhelmed. I returned to the office.

They say nothing in life is certain except death and taxes. I differ. Crisis is a sure thing. By definition, life is a series of crises, and a showcase of our ability to react. Death, sickness, raising teenage daughters….

Life and career choices determine the number and severity of crises you might experience. But one thing remains true: you will have them. Great leadership will minimize the volume of crisis, but every leader will encounter one. Preparation and execution determines how healthily you emerge.

No course, audit, or survey can tell you as much about your leadership than a crisis. If you want a test that shows what you are made of, crisis will reveal your abilities. Those who aspire for greater responsibility must understand that to whom much is given, much is required. The higher your position — be it family, church, community, or work — the higher the probability that you will be leading in crisis. Be prepared.

I have mishandled some crises and led well through others. In each case, I came to terms with my abilities. Failures and successes totaled, here are things I learned. Master these so they become part of your core leadership abilities.

Take Responsibility Immediately

Do not blame a vendor or an employee. You are the CIO. Crisis happened on your watch. Take responsibility and focus on resolution.


  • Chain of Command. Ensure everyone knows chain of command, especially when multiple teams are involved working on solutions. Given sleep cycles, you do not want lack of clarity to slow progress.
  • Battlefield Promotions. Expend your energy working with the motivated, not trying to motivate the worker. Make on-the-spot promotions as needed. Now is not the time for staff development.
  • Fit Leader. Sometimes a crisis can span multiple days. You have to be fit to be effective. Don’t argue with me, argue with science. Most can perform well for 24 hours, but notable performance degradation begins thereafter.
  • Visibility. You must be on site. Make a point to lead all customer calls (except on sleep rotation) and walk the floors of impacted hospitals. Walking floors is mandatory for all the command center commanders (my directs).
  • Deploy Listening Posts. During a crisis, it may appear that the sky is falling. You’ll hear exaggerated reports. Your plans will be incongruent with reality and spread panic and fear. Having your own listening posts will help discern reality and lead to quicker resolution. Another reason why personally walking the floors is critical.
  • Ask the Right Questions. We live in an instant society with on-demand entertainment and microwave food. We often don’t have all the pieces necessary to solve a problem that might arise. The delta between the immediate need for an answer and the time it takes to find the right solution frequently generates stress. In this scenario, stress begins to ebb when you finally start asking the right questions and start getting the right answers. And, like any good jigsaw puzzle, the pieces naturally begin to fit together… as they were intended to.


  • Operations. I am most familiar with ITIL. The operational process you choose to leverage is immaterial, but having established and routine processes is a key success factor during a crisis. You do not have time to reinvent the wheel.
  • Downtime Procedures. Again, establish and practice.
  • Disaster Recovery/Business Continuity. Most organizations have a DR plan, but few have BC drills. Conduct BC drills quarterly. This enables you and your staff to better handle the stress and drama of an actual crisis before it happens.
  • System Access. Avoid single points of failure. In an emerging world of ubiquitous electronic health records, you must have devices and systems pre-deployed to ensure access to data in a catastrophe.
  • Business Resumption Plan. While key to focus on solutions, you must also direct your staff and customers on business resumption planning well before the solution is in place.

Practical Logistics

  • Food. Assign someone to ensure a steady food and coffee supply. Let your key people focus on tasks, not noisy bellies.
  • Sleep. Have a rotation for rest, like airline pilots on international flights. Have comfortable places for people to sleep and nap if staying on premise.
  • Command Center. Stand up a center within one hour of calling a disaster and staff it 24/7. Should stay open 2x length of actual event. Do not shut down prematurely.
  • Assist Customers Impacted. Constantly ask, “How can we serve you? What else can we do?” whether IT related or not. I deployed staff to delivering water supplies and purchasing fans. Double or triple the number of staff on site. Visibility in crisis is crucial. Keep high staffing levels until the customer signals enough. I saw firsthand how our clinicians reacted to seeing a significant presence on the floors with questions like "How is the system working? How can I help?" This reassured our clinicians that we were taking the crisis seriously.
  • Communications Plan. Strong communication fills the void that otherwise gets populated with incorrect messages. Helps develop customer allies in solving crises, as opposed to antagonists. Publish your cell phone number. Start all communications by highlighting your organization’s mission. This serves as a common rally point for all involved. Be consistent in your messaging. Key messages might include accountability, transparency, action, calm, and hope. Execute your plan as published. Leverage multiple venues such as conference calls, e-mails, collaboration tools, portals, etc. Embrace corporate communications. They are experts in communications and can help you develop, adjust, and execute your communications plan.

Profit from Crisis

Document throughout, and take history of all actions and issues. This is critical in averting future crises. Resist the pressure to return an organization to status quo so you can profit from the crisis. Not seeking opportunities or pursuing the underlying cause of the crisis might leave your organization open for future conflict.

  • Wiki. Open a wiki and encourage staff and customers to post notes real time. Use these for practical insights during the crisis to document key lessons learned.
  • Document Lessons Learned. Encourage all customers to take notes during the crisis so they can make adjustments to the processes.
  • Downtime Procedures. These may never have been exercised. The best time to make real world adjustments is while downtime procedures are active.

Engage Outside of IT

  • External Expertise. It is a sign of strength to reach outside of your organization for help. If I sense the crisis is longer than two hours, I am on the phone calling peers and vendors.
  • Guru Council. Set up a council of advisors to make sure your plans are logical and nothing is missing. Council members are not in the heat of the battle and can provide unstressed ideas.
  • Vendor Management. Do not hesitate to escalate early and often. You have no time to dally. Let the level of severity determine when to go to the CEO.
  • Engage Senior Leadership. Do not hide what is happening. Engage senior leadership immediately and keep them informed. Bring senior leadership directly into the loop with vendor senior management. This ensures your crisis will receive appropriate attention.


  • Take care of your staff. Keep everyone focused on solutions not blame. Share all positive feedback as received.
  • Have multiple teams working on multiple solutions. On two occasions, the primary plan failed to bring about resolution. Fortunately, secondary plans already underway saved the day.
  • Ask for ideas from staff not associated with the crisis.
  • Levity. Despite the crisis, you must work hard to ensure a calm atmosphere. Staff will think more clearly when you de-stress the environment. I recall Day Two of a crisis when someone began playing Christmas music and a sing-along started. It alleviated an otherwise tense situation.

Ending Well

When the crisis is over, the work begins.

  • Send a Thank You. Personally acknowledge all those impacted, first your customers and then your staff. These might include nurses, medical staff, and practices.
  • Root Cause Analysis (RCA). Figure out what happened and what can be done to avoid this same crisis. Do not skip this. Publish the RCA and include action and mitigation steps. Monitor for execution.
  • Assimilate all lessons learned, downtime procedure modifications, etc, into enhanced processes.

We are all healthcare IT leaders, and my hope is that some might profit from the ideas posted. What ideas and tips do you have that I failed to cover? We will send a “crisis agenda” template to all those who post a new idea.

Ed Marx is a CIO currently working for a large integrated health system. Ed encourages your interaction through this blog. Add a comment by clicking the link at the bottom of this post. You can also connect with him directly through his profile pages on social networking sites LinkedIn and Facebook and you can follow him via Twitter — user name marxists.

HIStalk Featured Sponsors


Currently there are "20 comments" on this Article:

  1. People are good for 24 hours straight? I remember taking 4 minutes + or so to key in a frequency during a conversion (long before we had imports) after 20 hours of having been up. While I was up a couple of years ago for some 40 hours for an interface go-live, I never want to repeat this in my life. 24 hours is just too long for many of us, particularly if the crisis is going to go on for more than a day.

    This, by the way, is an interesting article about the effect of sleep deprivation: http://isme.tamu.edu/JSCOPE97/Belenky97/Belenky97.htm

  2. Be Calm. In the first hours of a crisis, everyone will look to the leader to set the tone. Having worked through contingency plans in advance provides the foundation for a sincere air of confidence which it critical. But even in the absence of actually feeling confident, the leader must project a calm confidence. Take a few minutes early to put on your poker face if you have to.

  3. Ed, could not agree more on your “lessons learned”. Our organization has practiced many of recommendations and it was the CIO who set the tone. If he/she is calm, focused and “in charge”, the resources will follow. Team work to resolve problems is a beautiful thing to watch in action.

    From a technical perspective, open a conference bridge immediately. Everyone gets on and no side conversations. Everyone will benefit from hearing discussion.

  4. Great post summarizing a great game plan. I’m inclined to feel a bit bad for Mr. Marx though, because many of these lessons are only learned through multiple events. If there is a Doctorate in Crisis Management, he qualifies.

  5. Great article. The most comprehensive, accurate set of steps to follow I’ve ever seen. Obviously he’s been there and done that. I’ve been through two crises – handled one and was a star, stumbled on another where Mr Marx’s list would have saved me. Thanks – an article worth saving.

  6. Great article. My two cents worth: Assume Murphy’s Law is in effect, and avoid the Alexander Haig “I am in charge” mistake! When the 35W bridge collapsed in Minneapolis, our data center lost power temporarily, we received many of the injured patients, cell phone circuits were totally overwhelmed, and our CIO was half way across the country on vacation. Chain of Command was confusing, and communication was almost non-existant for a period of time. We pulled through, and had a thorough post-incident review afterwards.

  7. Ed,

    Would the beginning of the crisis be the best time to fire and rehire yourself, so you head in to the crisis with a fresh outlook and motivation?

    Or would firing yourself after the crisis is over be better, to show you are taking responsibility and to start afresh?

  8. You never know what people are going through..this is a great template for an action plan, and is sensitive to all included. Kudos Ed.

  9. Thanks

    I like drawing the analogy to being a captain on a ship. When disaster strikes, you have no other choice than to stay calm, think clearly, stay confident, and address each issue until the crisis is over.

    And forging personal relationships with the crew prior to the crisis enables the team to work as one.

  10. I think you also need some type of tracking system to track the issue, its status, and the resolution. Whether it be an online system (ideally) or a spreadsheet, the information needs to be documented.

  11. Ed
    Great advice. !! Along with to whom much is given, much is required there is a partnering phrase: listen much and speak little. There are many that need to be heard but if the crisis is escalated to high emotions – some of the best leadership is just to take the time to listen. They can’t hear what you have to say until they know you have really listened to what they are saying however exagerated, when they are at their highest crisis point. That is an art I see rarely practiced
    Keep your blog going we all need to hear the wisdom

  12. Good collection of best practices. Thanks for sharing. Another idea is to demonstrate or articulate sympathy in your messaging. Avoid saying “I know what you mean” or “This hurts me as much as you”. Instead convey that while you may not know the extent of the negative impact the crisis is causing to them, you know that it is a negative impact and it is troublesome and why you are doing everything possible now to get a fix in place….

    Interested in your template.

  13. Ed,

    Good post. Having been through several disasters as both leader and staff, your points ring true. Thanks for nicely collecting and organizing the thoughts.

    I would suggest you consider amending Visibility to include the Leader(s) being visible in a meaningful way to their staff also, while of course being careful not to get in the way of the actual remedial work being done.

    I’d also suggest being sure to have complete and currect contact information for the entire IT team and customer leadership. Communication trees can also be helpful to faciliate information sharing and ensuring all the right folks are being kept informed — especially for leaders who became that via battlefield promotion.

  14. Wait, I didn’t see “stockpile MRE’s” in the food section 🙂

    It’s clear where the crisis management experience was learned early….

  15. Ed, great points to follow during a crisis: take responsibility (#1), establish a chain of command, listen and communicate. You bring up an important and sometimes overlooked point of being fit so a team (and leadership) can operate to the best of their ability in a crisis. Inadequate sleep and food will needlessly tax the ability of the team to bring the best possible results when needed most.

    One of your readers offered a link to an interesting military sleep study. As a side note, this reminded me of a number of astraunaut studies, including the effects on performance of high cognitive workload and sleep restriction: http://www.med.upenn.edu/uep/projects_nasa.html

    One of many, this study points also to developing bio tools to spot fatigue (these include a visual recognition of emotional expressions i.e., happy vs sad vs anxious, slow blinking eyes, etc), how astronauts can self monitor, and how to counteract the effects. A number of additional studies point to how to maximize when and how much sleep (naps) can counteract the effects, and how to maximize sleep. These all point to the critical importance of simple things like food and rest during a crisis event – any time optimal performance is needed.

    Thanks for sharing this!

  16. Ed, Hats off to you. Great article !! Definitely for sure the strategies and assumptions you have mentioned in here are a great learning indeed. Not only external natural incidents that take place outside the organization greatly impact the organizational well being but also internal incidents like system failures and system unresponsiveness can bring catastrophic situations in terms of healthcare delivery and patient care. Taking good case examples from the recent events like “tylenol poisoning case” and how Johnson and Johnson handled the whole situation to retain the consumers confidence. Or way back to 1979 incident at “Three Mile Island “ which gave rise to evolution of “Crisis Management planning” . Its proved very well that organizations cannot avoid a crisis . As you said “Crisis is sure thing”. Also a key fact that emerged out successfully is that crisis does not only mean danger. It also meant an opportunity for the organizations to identify their loopholes and find out means to protect themselves.
    One of the initiatives I would like to see in my organization from the leadership point of view is to “ Learn from other’s mistakes”. The process would be to collect and analyze the case studies of various healthcare organizations that have lived through crises. These case studies can help any organization actively analyze and critique the situations and strategies followed by the victim organizations to combat the crisis. It could act like a blueprint for any organization helping them in responding to similar situations and attacks more accurately and effectively right on time. Discussing the scenarios with your Crisis management expert panel time to time will avoid lot of mishaps at the time of crisis.

Founding Sponsors


Platinum Sponsors




















































Gold Sponsors












Reader Comments

  • Nasty Parts: I agree on the Olive posting. They have gone through a number of sales execs there. Of the ones there, none of them se...
  • Dan: It’s also not “logical” for French people to be mispronouncing Latin so badly that it turned into another language...
  • Brian Too: Yeah, but the nickname for Sault Ste. Marie is "The Soo", and not "The Sue". That just raises other thorny issues. T...
  • Brian Too: I have a completely different mindset regarding COVID vaccine supply shortfalls. Not from you Dr. Jayne, at least not t...
  • Left Coast IT Guy: Building effective disaster recovery and highly reliable systems both require a high-availability mindset. I worked in a...

Sponsor Quick Links