Why You’re Doing Customer Research All Wrong
Companies of all sizes spend inordinate amounts of time and effort in testing their marketing: names, packaging, concepts, feature lists, colors, flavors... you name it, you can test it.
So I was quite surprised when Jeff Henning, CMO at Affinnova and an expert in market research (and an old friend), passed along a recent study his company did. The conclusion? Most of that product research gets only average results in the market. By and large, you would do just as well flipping a coin.
The problems start with brainstorming
Affinnova studied 100 testing campaigns that its clients had done in the past. Typically the testing process went like this: A company came up with a long list of potential ideas to test, whittled it down using mostly executives' intuition, and then tested the much shorter list of ideas. Affinnova, on the other hand, took the initial brainstorming list and tested everything on it, presenting the ideas in groups and asking participants to select their favorites.
The result? Customers often preferred the ideas that never made it onto the final short list for testing. On the average, companies only anticipated what customers would choose about half the time. In other words, they had collective marketing IQs of exactly 100.
The problem, Henning says, is the transition from brainstorming to actual testing. "In brainstorming, you need to come up with a lot of ideas," Henning says. "They're fine with that. But then they'll look at that list and say, 'We can't test all these ideas.'"
And most of the time they are correct. A market survey or focus group can only handle a handful of choices for participants—four or five at the most. "People look at the list and say, 'We have to limit it to the ones that are best.' They base it on their gut instincts."
Unfortunately, gut instincts are often wrong, no matter how sure executives are that they know their customers inside and out.
The divergence between gut and reality can come from a variety of reasons. You may trust what you've learned from your interactions with customers. But those may have happened years ago with inclinations and market conditions changing in the meantime. You might substitute your own preferences for those of the average customer, assuming them to be the same when, in fact, they can be far apart. The mix of customers might have shifted over time and altered responses.
What makes the coin toss even more dangerous is that average is just that—average. Sometimes the results will hit one out of the park, and that's what you remember. But the next time could just as easily bomb.
The right way to test
The way out, according to Henning (not too surprising, as this is what his company does), is to optimize the choices before going into other forms of market testing. By showing groups of options to consumers and letting them go through one set after another, making actual choices, it's possible to get an optimized list of as many as 10,000 variations with as few as 450 subjects from online panels. Smaller numbers of variations can be tested more quickly, but the important point is that intuition dramatically fails.
"We did a beverage study where the client was really convinced that the length of the bottle neck was important," Henning says. But tests showed that neck length was not a driver for consumer choices. Similarly, an ice cream company tested variations of ingredients like fruits, nuts, or candies mixed in with various flavors of ice cream. Executives expected that some combination of included goodies would win out. What did? Plain vanilla and strawberry, types of flavors they hadn't considered.
In the test of previous concepts, Affinnova found that optimizing lists in advance resulted in 60 percent improvement in performance. A study of product launches from 2003 to 2008, comparing optimized against non-optimized concepts, suggested that optimized products were more likely to be on the market after two years and would earn 2.6 times the revenue.
That's almost enough to make you want to stop wetting your finger and holding it up in a breeze to tell the weather.
ERIK SHERMAN | Columnist
Erik Sherman's work has appeared in such publications as The Wall Street Journal, The New York Times Magazine, and Fortune. He also blogs for CBS MoneyWatch.