3 Studies That Prove You Shouldn't Believe Everything You Read
BY Erik Sherman
There is no shortage of studies promoting theories on how to run your business better. But don't be fooled: not all research is created equal.
Smart entrepreneurs look for information and knowledge to better run their businesses. One source to be au courant should be contemporary studies and surveys that test assumptions, explore relationships, and help refine how people look at running businesses.
With a setup like that, you know there's going to be a "but." And there is--a big one. You have to be on your toes when looking at studies. Not only are results often over-hyped by companies using them as a way to indirectly promote themselves, but the way they were conducted can undermine any credibility they hoped for, even when performed by academics.
Here are three such studies that show you might not want to believe everything you read.
Finding #1: Surprise raises make people work harder.
The academics used global freelance job board oDesk to hire 266 people for a supposed data entry job. One group was told they would be paid $3 an hour. Another group was started at $3 an hour but then told before the project began that the rate had gone up to $4. The third group received $4 an hour from the beginning.
What they found was that the second group worked harder than the third, even though the two were ultimately paid the same. The results showed a roughly 20 percent higher productivity rate for the group that got the surprise bonus than the two that were paid either $3 or $4 from the start. Productivity was even higher among employees who had the most experience--perhaps their experience allowed them to work faster, or they knew enough to be impressed by the surprise gift.
According to HBS professor Deepak Malhotra, how you pay people is just as important as what you pay. But hold on a minute: $3 and $4 an hour? Yup, all the people were likely from overseas. With 266 subjects total, there were only enough to provide 88 or 89 people for each wage group. Most importantly, there seemed to be no control for cultural differences, which could conceivably be the largest factor in responding to a reward system. Given the small size of the sample and the lack of cultural information, this is a study that can tell you exactly nothing about how any market, let alone western workers, would react to compensation structures.
For the first-of-its-kind study, the researcher used survey data from four U.S. colleges to examine how students' public-welfare beliefs change during their college engineering education and whether the curricular emphases of their engineering programs are related to students' beliefs about public welfare. The study found that engineering students leave college less concerned about public welfare than when they entered.
The study included more than 300 students who entered engineering programs as freshmen in 2003. They were surveyed periodically over time up to 18 months after graduation about professional and ethical responsibilities. But that means Cech spoke only to the engineering students, not a control group. For all she knows, everyone going to college goes through this. The study used as unscientific an approach as you could possibly imagine. (Tip of the hat to Quartz for having noticed the study and its fundamental flaw.)
Finding #3: Knowing an entrepreneur makes you more likely to become one.
The last example (not for lack of material) is a study conducted at the Ewing Marion Kauffman Foundation by Paul Kedrosky, a venture capitalist and senior fellow. He wanted to know whether there is a viral aspect to being an entrepreneur: that is, if you know a lot of entrepreneurs, you're more likely to become one.
Kedrosky created a pair of two-question surveys and offered them to a "national sample" of 2,000 people. Depending on the question, responses varied from 400 to 2,000. That variation and the lack of any more information on how the people were solicited suggest that the results should be taken with salt. Lots of it.
For one thing, he asked people whether they knew an entrepreneur. I'd argue that most people in the country--remember, it's supposed to be a representative sample--would more likely respond to the term "someone who owns or started a business." Without that more recognizable term, it could be that many more people would have said that they knew at least one entrepreneur, and the apparent correlation between knowing an entrepreneur and being one might have disappears. And that's without questioning how, after partitioning answers, the data samples were of people who knew entrepreneurs versus people who thought they didn't. (Let's not even touch on what percentage of respondents were actually "entrepreneurial wannabes.")
When it comes to examining studies, be skeptical. Even the pros can play fast and loose with research.