A couple of weeks ago, we performed a major upgrade on our ambulatory system. Officially we’re now ready for both Meaningful Use Stage 2 and ICD-10, with all the bells and whistles installed. As upgrades go, this wasn’t my first rodeo. It went smoothly with only one minor IT concern and no significant incidents for the end users.
Since no good deed should go unpunished, management is now looking to cut our personnel resources for the next one. They can’t seem to understand why several hundred hours of work went into the upgrade because clearly it was “no big deal.” Mind you, these are not old-school IT managers, but members of our ambulatory operations team who want to avoid having super users out of the office.
We rely on the participation of super users, not only from the ambulatory practices, but also from our central business office, central scheduling department, and central referrals department. No one knows end user workflows like the super users who work with them day in and day out. We have detailed test scripts for our internal testing, but we need real-world expertise to tease out the smallest bugs. Like any organization, our users have some creative workflows that we don’t train, and if we don’t have their participation, we won’t find those issues until go-live.
We’ve been using the same upgrade methodology for half a decade, which is usually goes off without a hitch. It’s a belt-and-suspenders approach, with some duct tape and baling wire thrown in for good measure. We do a dry-run upgrade just prior to the super user testing so that we can get our timing down pat for the main event. The upgrade weekend playbook has some elements timed to the minute and there is a single upgrade commander responsible for ensuring every step is completed and communicated.
Because of the need to involve a couple of third-party vendors to handle some data migrations that we wanted to perform while we had the system down, timing for this one was even more critical. There were numerous handoffs among DBAs, access management, application analysts, build analysts, internal testers, and end-user smoke testers in addition to the third parties. Although we don’t make everyone sit on a bridge line and talk through their work and the hand-offs, we do require people to notify the team when they complete a step or if they’re running behind so that we can adjust if necessary.
The lead analyst that usually quarterbacks our upgrades had an unexpected medical issue a handful of hours before we were due to take the system down, so I ended up co-managing it with one of our analysts. This meant being on call overnight for issues, which doesn’t bother me. Once you’ve been on trauma call or managed an ICU full of patients overnight, being on upgrade call doesn’t seem very scary. Still, you never want to hear that phone ring in the middle of the night. Shortly after midnight, I decided to grab some sleep since we weren’t expecting a handoff until early morning.
When the phone rang at 3 a.m., my heart was pounding. The tone in the tech’s voice wasn’t reassuring as she apologized for calling. Apparently the upgrade was running nearly three hours ahead and she wasn’t sure if she should wake someone up to tell them or not. I have to say, seeing an upgrade run ahead, especially by that much, isn’t something you see every day. I shuffled out of bed and we walked through the checklists to make sure nothing had been missed. I cruised the error logs as well. Nothing was amiss, so we had to chalk it up to the production server being faster than our test platform.
We must have our share of either insomniacs or nervous Nellies on our team because a couple of people were showing available on our instant messenger service. They were happy to launch the next few steps early. Despite the call being a non-issue, once your adrenaline is flowing, it’s hard to get back to sleep. I curled up on the sofa with some journal articles, which thankfully did the job. By our 8 a.m. status call, I was rested up and eager for the build and testing teams to get to work.
Even though everyone has remote capabilities, we require the regression testers and analysts to be on site. We’ve learned the hard way that people are sometimes less attentive when working remote on the weekends. Sometimes it’s just better to have two sets of eyes looking at the same screen together (without a WebEx lag or dogs barking in the background) for troubleshooting. It’s a sacrifice for the team to come in, but we try to make it as fun as possible. The kind of team-building you get from an event like this is often priceless. It’s also important for the end user and analyst teams to work closely together and build mutual respect.
In response to the questions about why we spend so many hours preparing and delivering an upgrade, I’m going back through the last couple of months and highlighting some key milestones that may have been riskier with a leaner team. We have multiple people trained to do each task, which was clearly helpful when our quarterback unexpectedly sat out the game. I’m also working to quantify the intangible benefits of having disparate teams work together.
We ended up being able to re-launch the system two and a half hours early, which meant less downtime procedural re-work for the patient care sites that are open on weekends. Due to the diligent prep, we also had fewer phone calls Monday morning than we’ve ever had. That’s got to be worth something as well. The question is whether the Administralians will agree with our analysis. If they don’t, maybe we can let them run the next one and see what happens. We’ve already documented our lessons learned and updated the project plan, so it’s ready to ride.
Ever jumped in when someone said “Cowboy up?” Email me.
Email Dr. Jayne.