The article cites research that people tend to be more creative in the presence of noise than in complete silence and that EKG readings reveal that "a certain level of white noise proved the ideal background sound for creative tasks."
Considering that white noise is fine and even beneficial,
Why do so many of us hate our open offices? The quiet chatter of colleagues and the gentle thrum of the HVAC should help us focus. The problem may be that, in our offices, we can't stop ourselves from getting drawn into others' conversations or from being interrupted while we're trying to focus. Indeed, the EEG researchers found that face-to-face interactions, conversations, and other disruptions negatively affect the creative process. By contrast, a coworking space or a coffee shop provides a certain level of ambient noise while also providing freedom from interruptions.
I think that's true as far as it goes, but there's more going on here than meets the ear. Coffee shops (and co-working spaces) operate under different social systems than offices and workplaces.
For example, in a coffee shop, all patrons are created equal. There is significant social pressure to adhere to common-sense politeness, such as no telephone calls (you take it outside) and no loud talking.
By contrast, in an office, social pressure exists within an informal hierarchy of clout. While there may be rules--even posted rules--similar to the unspoken rules of coffee shop behavior, breaking those rules--and escaping censure for doing so--is a highly visible and universally understood way to establish dominance.
Indeed, sexual harassment (especially when committed in the presence of underlings, as has been the case in many of the recent horror stories) is the most extreme form of this kind of status-establishing bad behavior.
Less heinous, though equally status-driven, behaviors plague most workplaces. For example, in a high tech firm, a project leader might (consciously or subconsciously) hold a loud conversation in the middle of a shared work area simply to prove to everyone else that his project is more important than whatever they're doing.
Is he being a jerk? Sure. But while people may resent it, if he has enough political clout, not only will he get away with it, but the fact he can get away with it will emphasize and reinforce his status as well.
Another difference between coffee shops and open-plan offices is the nature of the conversations that you're likely to overhear.
In a coffee shop, there's an infinitesimal likelihood that an overheard conversation will be relevant to you or your job. By contrast, in an open-plan office, any conversation is potentially relevant. As a result, your brain is going to keep half an ear cocked when anybody is talking.
Another big difference is your level of control. If you're in a coffee shop and find the noise distracting, you can wear noise-canceling headphones with a reasonable expectation that nobody will ask you to remove them.
In an open-plan office, though, other people (especially those who believe they're more important than you) feel empowered to catch your eye and demand your attention. They may even believe they're doing the company a favor by pulling you away from your playlist and back into the real world.
In short, it's not the noise that makes an open-plan office such a miserable place to work--it's the inability to escape the proximity of the petty and annoying behaviors of your co-workers.
One point that the HBR article entirely missed was that there's a perfectly reasonable alternative to both open-plan offices and coffee shops/shared work areas: working from home and using tools like Skype and Slack to control your interactions with your co-workers.
A personal victory.
While I'm on the subject, I'll share a real-life experience of how I dealt with some status-driven annoying behavior back in the day.
Early in my career, I worked for a company where everyone, even the C-level execs, had cubicles. While I found the environment distracting, I made the best of it by requesting a cubicle off the beaten path, next to two cubicles that were reserved for visitors, and therefore usually empty.
One day, however, a salesman was using one of the visitor cubicles to make cold calls. (This was back before cell phones, so the only way to make calls was on a wired handset.) Normally, I'm sympathetic to salespeople, but this guy was one of those fast-talking loudmouths who uses the same lame script over and over and over.
After about an hour of this, I'm about to lose my mind, so I pop my head over the partition and ask, politely, could he please keep it down. He gets all in my face about how his job is more important than mine and ends the conversation with a suggestion that I commit an unnatural act upon myself.
Fine, I say to myself.
I wait until he visits the restroom, go into his cubicle, unscrew the handset mouthpiece, remove the microphone, and replace the mouthpiece. When he returns and resumes cold calling, everyone hangs up on him because they can't hear anything he's saying.
He eventually gets so frustrated that he calls technical support, who, of course, also hangs up on him while he's explaining the problem. Cursing, he goes to find somebody who can fix the phone.
While he's gone, I replace the microphone.
In about 15 minutes, he returns with a support engineer, who tests the phone, confirms it's working properly, and then leaves, making it clear (in a voice of belabored patience) that he thinks the salesman is incompetent, crazy, or both.
By this time, it's lunch hour. The salesman, still fuming, heads toward the cafeteria.
I go back into the cubicle and remove the microphone again
He comes back and resumes cold calling. Soon he's so frustrated that smoke is practically coming out of his ears while I'm maintaining an innocent expression, all the while mentally ROFLMAO.
Finally, the salesman storms out of the building, never to return.
And I go back to work, with the satisfaction of a job well done.