When Bad Things Happen to Good Companies
Companies spend billions of dollars to protect their computer systems from attacks. Yet they still happen. Recently, one of those incidents hit dangerously close to home, when a company called GitHub announced that some of its users had been hacked.
GitHub hosts computer source code (what software is made of) for thousands of companies--including the source code for our products at 37signals.
The people who run GitHub are friends of ours. I really felt for them when they were forced to announce to their customers that their accounts may have been compromised. I imagined myself having to write a similar letter to my customers. The very thought made me queasy.
But it also woke me up and gave me an idea.
We are always working to improve our security. But I wanted our entire company--not just the tech staff--to figure out how to respond if we were hacked.
Lots of companies stage drills to improve their technical capabilities. They run load tests to simulate their systems being under enormous pressure. They disconnect servers to see what happens when a piece of hardware goes offline. They force their software to do things it wasn't intended to do to see what happens.
But what about the public-facing part of the company? Technical proficiency is important, but it's only half the battle. The other half is how your company communicates with customers when things go seriously wrong.
So after hearing about the GitHub breach, I gathered the entire staff. Here is what I told everyone: "Every time we read about a security breach at another company, I want us to act as if it had just happened to us. How would we handle it? What would we tell our customers? What would we do about it so it does not happen again? What steps would we take to regain customers' trust?"
We started right away. We studied GitHub's response and came away impressed. GitHub did an outstanding job of letting the public know what happened, how it happened, how to learn if a company had been affected, what GitHub did to halt the attack, and what it will be doing to prevent another one. (You can read GitHub's response here.)
The point person on our security team wrote a draft of what our response would have been, which was posted to a Basecamp project and circulated companywide. Everyone had a chance to suggest tweaks, make edits, and post comments.
Like GitHub, we tried to be honest, open, and informative. Anything less threatens to erode trust fast, especially when dealing with something as critical as data security. Writing the letter even revealed a few weak spots in our defenses, which we're working on tightening up. As often happens, the simple act of communicating helped us see areas for improvement we hadn't noticed before.
It was a sobering experience. I came away feeling even more sympathy toward the companies and customers who have been victimized by hackers. And it reminded me just how important it is to stay vigilant.
Next time around, I'd like to take it beyond our customers. It would be good to drill how we'd talk to the press or answer questions on Twitter. Meanwhile, the letter we drafted sits safely in Basecamp. I hope no one will ever have to read it.
Update: This column has been updated to clarify what happened at GitHub.