I’m writing this from the waiting room of an auto body shop. I had a bit of a disaster on Sunday, when my left side mirror and a post in my garage got a bit up-close and personal. After I dropped a few curse words inappropriate for television, I grabbed the groceries from the car and started planning. I had to plan two things: how to deal with the situation, and how to tell my husband that I had damaged our one and only car.
This small disaster reminded me of the many disasters I’ve experienced at work – both big and small. And how each of these disasters had the same four components: the disaster, the reaction, the recovery plan, and the communication.
Here are a couple of my harrowing tales, what I did right (and wrong!) in each case, and some hopefully useful takeaways.
My first career was in Information Technology. This means that I have some experience with technology disasters. Specifically, two back-to-back disasters involving many gallons of water and a server room. This earned me the nickname “Waterfall Girl” a few years ago. (Which didn’t really stick, luckily.)
The Disaster. Our water-cooled air conditioning that was right above all of our critical servers essentially burst. Computers and water don’t really mix well, so you can imagine.
The Reaction. I received a phone call on a Saturday night from our office management saying, “Hey; there’s a bit of a leak in the server room.” I called my systems administrator & he said he’d go over and take a look after dinner. The second call was from someone in the office, saying, “Jenn, it’s raining in the server room and there’s a couple of inches of water on the floor.” I called my systems administrator back and said, “Put down your fork.” Then I sped into the office.
The Recovery Plan. We knew we had to dry everything out, so we took apart all of the equipment & set it out all over the office. We planned to return 24 hours later and put it back together. And, except for a table collapsing while I was on top of it and my belated comment of, “Oh, hey. Where’s the fire extinguisher?” as I was plugging something back in, we got back up and running 36 hours after the waterfall.
The Communication. This one was difficult. See, we were a Blackberry-only firm, and the email and Blackberry servers were in pieces in the kitchen. Luckily, our firm administrator had a list of home phone numbers for the partnership. While we were taking the servers apart, she was calling the partners. We also printed out signs to post on the entry doors so that anyone coming in to work on Sunday would know that it was basically useless.
The aftermath of this one was two-fold and somewhat interesting. The first bit is that, while we managed to avoid the ‘blame game’ during the disaster, some of the firm’s leadership wanted to know who to blame for constructing the server room in the way that had the water above the servers. I couldn’t really answer the question – during construction, the partnership had actually denied my request to add space to the other side of the server room so that I could move the servers!
The second bit is that we had another waterfall happen again 5 days later after the maintenance company said they were going to fix the air conditioning “better than new”. My reaction to that one was a bit different and involved a few more words that weren’t appropriate for television.
Technology can be both a curse and a blessing. Disaster #2 was a different kind of leak – the kind that can be a Public Relations (PR) disaster. There was some important news that was going to go public in a week, but thanks to a technology mix-up, went live on the website much earlier.
The Disaster. The firm’s announcement went live on an overnight auto-publish of the marketing website. A few people picked it up on Twitter and the news was well and truly out.
The Reaction. I’ll admit that I didn’t react as well to this one as I did to the waterfalls. I heard about it via a tweet from a friend, but when I went to look at the website, nothing was there (IT had rolled it back when they realized what happened). I panicked a bit, thinking that I might not have a job in the morning. So I called my boss a few times and worried all night.
The Recovery Plan & the Communication. The information had already been pulled back, so there were two aspects of the recovery plan that also both involved communication: managing our higher-ups (who may or may not have been annoyed about the leak) and managing the PR around it. Aside from a few tweets, no other PR issues came up, but we alerted our PR team so they could keep an eye on it. The higher-ups were more problematic – we didn’t let one of them know in time, so she found out via Twitter and was UNHAPPY about not hearing earlier.
The aftermath of this one was that we put on an educational class for everyone who might make the same technology error that I had made (that caused the leak). Had this made the press rather than just Twitter, though, the recovery plan would have been much more extensive.
If you realize that every disaster at work – big or small – has these four components, you can approach things more methodically and with less immediate panic.
As my car can tell you, disasters happen. If you approach them methodically, you can get through with a minimum of pain and come out with a story that should earn you a beer or two when you tell it.
Best Companies To Work For