The concept of A/B testing is as simple as…well, A-B-C. Someone writes, say, a pair of headlines, sends them out into the world, and tracks which one snags the most eyeballs. It’s formulaic and binary--the digital equivalent of an optometrist asking you “number 1 or number 2?”--and has proven effective over the past couple decades as companies try to compete for attention in crowded markets.  

A/B testing has come a long way, however, and the practice has started to develop innovative branches. Today, clever firms are leveraging the knowledge they’ve gained from the digital world and are applying it to physical spaces in exciting new ways. Here is how companies are using A/B testing to attract customers, increase sales, and find solutions to stubborn problems. 

Data-Driven Testing

Retailers can learn a lot about you from your shopping habits, and any successful e-commerce site is constantly testing how layouts, button placement, and even colors attract customers. Physical stores, on the other hand, are trickier to transform on a minute-to-minute basis. That is, until recently. “Brick-and-mortar stores have been sort of a black hole for a lot of data,” says Scott Lachut, a partner at PSFK, a retail innovation research firm. “But now you’re seeing companies embed a lot of smart infrastructure into the physical space itself that [reveals] how people are interacting with the store by collecting data on customer behavior.”  

For instance, sensors placed in store ceilings provide store owners with a heat map, revealing where customers spend the most time. Providing in-store Wi-Fi can help merchants better understand how people use their phones while shopping. “By looking at the data they gather and moving merchandise around accordingly, [stakeholders] can optimize the in-store experience for the customer,” Lachut says.

Shifting best-selling items to display windows or putting brand-name products at eye level is nothing new in retail, but now retailers can easily point to data generated by in-store technology to make more informed decisions. “Instead of doing it by gut feeling, you can now look at the numbers and say, ‘What if we did this versus this?’” Lachut says. 

Make Experiments Count

Firms considering A/B testing in a physical space must do so rigorously, says Stefan Thomke, a professor at Harvard Business School and author of the upcoming book Experimentation Works: The Surprising Power of Business Experiments. Thomkebelieves that businesses need to be scientific when running A/B trials. “Many so-called ‘business experiments’ are not based on scientific and statistical methods,” he says. “Managers end up making bad decisions because they misinterpret noise as causation.”

Thomke cites the large national retailer Kohl’s as an example. In 2013, company executives reduced operating hours in certain stores as a test to cut operating costs; when sales suffered, some urged an immediate halt to the trial. But when analysts took a deeper look at the data, they found that even though the stores were open for fewer hours, the volume of transactions remained the same. When the average value per transaction returned to normal, the company achieved its goal of reducing costs while keeping revenue stable. It’s a winning example of trusting the data when running an A/B experiment.  

A/B testing is going through a renaissance, thanks to a generation of innovators who are bringing tech savvy to the traditional world of brick and mortar. Physical spaces are now looked at as products to be tinkered with, not final outcomes. Data has let decision makers experiment with purpose, improving and iterating in new ways. “The first idea isn’t always the most optimized,” says PSFK’s Lachut. “Now you can treat things as though they’re permanently in beta and tweak from there.”


A/B testing is just one of the advanced services Mailchimp offers in its all-in-one marketing platform. Find out more here.