The Problem With Off-hours Emergencies

I have a special affinity/interest for water treatment emergencies, so this article out of the New Orleans Times-Picayune about an unexpected drop in water pressure at a treatment facility due to a loss of power is, by itself, super interesting. Add in the angle about notification failures and the unnecessary delay of public information, and this might be one of the most instructive articles we’ve seen!

So, to start, I am not looking to indict the officials in New Orleans. Reading between the lines of the article, it looks like there was an alert, notification and information release plan in place before the crisis. And from the looks of things, most of the problems that occurred would not have happened during a planned exercise. Sometimes things go wrong, and the folks who went through it—and the rest of us—are obligated to learn from it. In that interest if you haven’t heard it before, let me be the first to tell you, test your alert, notification and public information release process unannounced and during non-work hours.

Now onto the meat and potatoes. At 10:30 on a Friday evening, the water authority noticed a dramatic water pressure drop at a treatment facility that serves more than 300,000 residents. Within minutes, the emergency plan was activated. Within 45 minutes, it was decided that there was enough of a pressure drop to require the issuance of a boil water advisory.

(Quick test: What does a boil water advisory say? Boil for one minute? Boil for five minutes? Boil drinking water? Boil all water? Boil toothbrushing water? Don’t drink shower water? There’s a lot more to a boil water advisory than many people understand. If you don’t have an approved template, get one.)

Just past midnight Saturday morning, officials from the New Orleans Office of Homeland Security and Public Safety (OHS & PS) were in contact with water authority’s (S&WB) emergency preparedness director about the situation. S&WB noted then that they were recommending that the public in the affected area boil their water. It wasn’t until 7:30 Saturday morning that OHS & PS acknowledged this message, and asked for a copy of the advisory. S&WB quickly forwarded a four-sentence description of the order, but still had not updated their website or issued a formal press release. OHS & PS forwarded information out to the public using their emergency alerting system around 8:00 Saturday morning, with apparent disregard for their alerting protocols, which dictated that nothing be sent out with an official copy of the advisory. That official advisory was sent out on Sunday evening. The article also goes into problems with getting in contact with senior administration officials in the Mayor’s office.

So, what can we learn from all of this? Tons.

New Orleans noted that during emergency situations, emails should be followed by phone calls to ensure receipt, which is a great start. I would take it a bit further, and since it read like OHS & PS was up and available at the beginning of this all, they should have played a sort-of “watch command” role, wherein it was their duty to ensure that everyone who needed to know that something was going on, knew. While it’s nice (and important) to make sure anyone who needs to contact those who need to know, if one person in that chain takes the night off, or turns their phone off, the whole chain of command breaks down. Assigning responsibility to a 24/7 entity to coordinate the notification process speeds ensures that the job will get done.

The public information release part of this is interesting, too. Why did it take almost two days for S&WB to issue a formal alert about the situation? Given that situations that can cause a boil water alert to be issued are among the most likely of emergencies at treatment facilities, one has to wonder why a template press release and official alert weren’t previously developed and approved. Something with <insert information about treatment facility here> and <insert date and time of event here>, but that still had the official recommendations ready to go would have saved hours, I imagine.

And S&WB isn’t the only party that should learn something from this situation. OHS & PS should update their alerting protocol to allow for unofficial communications to be released — exactly as they did in this situation. So, kudos to OHS & PS for understanding that the public’s need for information in an emergency can trump protocol, but now they need to understand the value of the flexibility demonstrated by the person running the alert system and enshrine that in the official protocols.

In sum, again, I don’t mean to pile on to the folks in New Orleans. I think that they had a great plan and followed it—for the most part—to a “T.” But emergencies rarely happen between 8:30 and 5:00, from Monday through Friday. They happen on Friday and Saturday nights, in northern states on the coldest night of the year, in the south on the swelteringest August afternoon, when that one guy that’s crucial to the response is on a cruise. So while it impossible—especially with today’s budget climate—to do any large-scale exercising off-hours, it’s cheap and easy to do a call-down in the evening, on the weekend, when somebody’s out of town, just to make sure that you can.

Thank you, New Orleans, for being just prepared enough to prevent any real problems, but just “like the rest of us” enough that we can all learn something from your experience.