"Thou must have good product/market fit." It's a mantra that's practically stamped on every entrepreneur's forehead. But how do you know if you have achieved good product/market fit? The answer: For many, it's by getting out of the building and conducting man-on-the-street interviews to validate your concept or minimum viable product.
Ideally, if a certain number of people say they want (or actually buy) your product or make it clear that they would be disappointed if it were no longer available, you have achieved good market/product fit. In fact, interviewing customers/prospects is the cornerstone of any new business and a key feature in every good startup weekend and customer development workshop.
While a well-intentioned notion, entrepreneurs conducting market research is--at best--deeply flawed. At worst, it can lead founders to make decisions based on really bad information gathered in a vacuum.
Welcome to the observation effect
The biggest problem with asking someone what they think of your service, app, or product is that your respondent is going to tell you what they think they feel versus what they actually feel. What people say and what they do are totally different things.
TED speaker Beau Lotto, professor of neuroscience at University College London and founder of Lottolab, puts it this way: "Focus groups are like theater. People behave in a way that they think they are supposed to."
"Think about the last time you were at a shoe store and the salesperson asked you to walk," says Michael Lee, head of executive planning at VCCP Partnership and speaker at Cannes Lions this summer "People tend to change the way they walk. They will break into a stride or swing their arms, basically they do things they never normally do because they know they are being observed."
Researcher Sarah Pearson proved this point empirically. She asked respondents to answer questions about their TV viewing habits. She then recorded them watching television. In an article in The Economist, Pearson writes, "There turns out to be an enormous gap between how people say they watch television and how they actually do. In surveys, they almost always underestimate how much television they watch, and greatly overstate the extent to which they watch video in any other form." The upshot? What people say is different then what they actually do.
The antidote for the observation effect: watch, don't ask
Pearson's study is a tip-off to the first solution. Watch what your customers do versus what they say. Good webmasters focused on usability have been practicing this tactic for years., "It's important to test users individually and let them solve any problems on their own," says technology consultant Jakob Nielson. "If you help them or direct their attention to any particular part of the screen, you have contaminated the test results."
The founder of beverage maker Bai, Ben Weiss, was able to watch his customers by volunteering for hundreds of in-store sample sessions at a big box retailer. Watching customers' reactions, and more importantly, whether they picked up an item and put it in their shopping cart, proved to be a great laboratory for learning.
Solicit feedback anonymously
If you're reviewing your concept or MVP with someone, have the person write down his view of the concept first. This will minimize bias in the answers. Another good method is to conduct online reviews of the product. Dropbox, for example, created a simple video of their MVP and posted it on Digg to get anonymous feedback from users, while Mighty Handle used anonymous feedback from Amazon reviews to improve its product.
Minimize focus group bias
If the only option you have is to interview customers yourself, here are a few things you can do to reduce bias:
- Moderator Bias--Everything from your body language, style of dress, race, and gender can affect bias. You obviously can't change your gender, but you can try to be as neutral as possible in terms of dress, body language, and style.
- Questions Bias--The way you ask a question can lead to a biased answer. The biggest point here is to not offer leading questions. A leading question suggests what the right answer should be. Here's an example: Some people think working out more than five hours a week can be bad for you. What is your opinion? Instead, reframe the question in a more neutral manner. "What is your opinion about people who work out more than five hours a week?"
- Participant Bias--Make sure you are talking to a customer set that truly represents your target. If you're targeting mothers and you only interview suburban moms with toddlers, that isn't representative of urban moms who have infants or single mothers with elementary school kids.
If pressed, ask the most important question
If the only way to solicit feedback is directly, start by asking a good question, Lotto suggests. "Who, what, where...questions are meaningless," he says. "The best question to ask is 'why.' Why does someone like the app, or service, or product. If you can begin to understand peoples' intentions, you can get at peoples' motivations, which will help you with understanding."
Getting customer feedback is critical. The key is to collect the info in ways that minimize bias so you can get the unvarnished truth. Weiss says it well: "You have to get out in front of consumers and test ideas. Ideas are only ideas until the consumer embraces them."