Flirting with Disaster
Strategies: Disaster Plans
It 's every CEO's worst nightmare, the corporate equivalent of a nuclear meltdown. A sudden, uncontrollable event paralyzes the business, wipes out crucial company data, and costs a fortune to fix. The surest way to stave off such terrors -- or at least mitigate their impact -- is to develop a disaster plan, a smart security step that nonetheless scares off altogether too many CEOs. Even among the brave ones who've faced that fearsome task, few have put their disaster plans to the test. That's crucial, as MSS Group (#135) discovered the hard way when the company found that its disaster plan was disastrously flawed.
It all started on a June night in 1998, when a programmer sat down to do some after-hours work. "Two hours into it, she realized that she had put in the wrong command, and the system was writing over all the customer data," remembers CEO John Podrovitz. "She started a chain reaction in our system."
The night manager phoned Podrovitz with the bad news, which only got worse. Although the company had hard copies of the lost data, no one knew how to upload the records except the director of technology, Cynthia Raber. And she was on a bicycling trip -- in Spain. "We had a system in place, but no one had ever tested it except for her," says Podrovitz. "We went into pseudo-panic mode."
Because the bulk of its business is real-time bill processing for telecom services, MSS needed to get the data back online right away. But by the time the company tracked down Raber, 35 hours had lapsed and MSS had lost track of thousands of transactions. Even as Raber talked a programmer through how to restore the system, MSS fell further behind. "We lost about five days of work," Podrovitz says.
What's more, the ensuing labor costs of retrieving lost customer data -- which entailed paying staff to work in 20-hour shifts and through weekends -- exceeded what MSS otherwise could have spent on hardware to house additional backup capacity. "We were penny foolish," Podrovitz says. "You can buy hardware, but you can't buy time."
Since then MSS has revamped its system. Instead of doing a nightly backup as before, the company now conducts partial backups every three hours and performs a total replication every five hours. To boot, each month Podrovitz picks a different technical staffer at random to run through a complete disaster-recovery drill. "We develop a real high-pressure scenario," Podrovitz says. "It's become a religion with us."
Entrepreneurs would also do well to shore up against catastrophes that come from outside the company. That's especially important in locales like Florida, where Sarasota-based LexJet Corp. (#175) has a long-standing contingency plan for surviving the region's six-month hurricane season.
LexJet -- a direct marketer of printing materials -- had its closest brush with disaster in September 1998. That's when Hurricane George bore down on Florida's Gulf Coast. By noon on a Friday, storm watchers were predicting a direct hit on Sarasota, and local officials ordered a mandatory evacuation. "We were anticipating that Monday morning we would be without utility service, without phone service, and maybe even without a building," recalls LexJet president and CEO Ron Simkins.
Simkins knew that LexJet's inventory of printing material was safe. Stashed in a warehouse in Milwaukee, the stock was well beyond the storm's reach. But because most of the company's customers required that their printing materials be delivered within one or two days, the business couldn't afford to shut down until the storm had passed. "We had to make sure that we could ship product and have people on the line for customer calls," explains LexJet vice-president Dean Lambert.
So while most Sarasotans simply headed for higher ground, LexJet deployed its disaster plan. Half a dozen of LexJet's 14 employees packed up the company's client-contact data and headed for the airport, where they prepared to board the first available flights for Atlanta, where LexJet had lined up call-center backup and short-term office space for just such an occasion. Meanwhile, Simkins sent an E-mail alerting LexJet's 2,500 customers about the hurricane warnings and letting them know that until further notice the company's calls would be routed to the service center and then forwarded to temporary offices 450 miles north of Sarasota.
As it turned out, LexJet's emergency team might as well have hunkered down in Sarasota. To everyone's surprise and relief, Hurricane George veered west toward Texas and then headed back out into the Gulf. By Monday morning, it was business as usual in Sarasota. Still, LexJet's managers say that this unanticipated trial run of their contingency plan was an important, and worthwhile, exercise. Says vice-president Lambert, "When the hurricane season comes around, people in Florida have to take it seriously."
Please e-mail your comments to firstname.lastname@example.org.
To learn more about the Inc. 500, visit the Inc. 500 area.