Pretty much all of the reviews on the accommodation booking site Airbnb are biased--or were, before the company started experimenting with its system.
And as it turns out, the bias skewed the tone of these ratings in what would seem to be a pretty surprising direction: They were too glowing.
But this didn't surprise Airbnb data scientist Dave Holtz. "The ratings in most reputation systems are overly positive," Holtz said Thursday in San Francisco, speaking at Import.io's Data Summit.
This is a problem for Airbnb, because its system relies on reputational data in order to function properly. For example, the company doesn't want to keep directing renters to a home with a mildewy bathroom and a shabby mattress as a result of other users' providing inaccurate reviews or none at all.
So how does the overly positive bias creep into the review system? It's all about psychology, Holtz explained during his talk entitled, "If You Don't Have Anything Nice to Say, Please Say Something." First, it's important to understand how the old review system worked.
Prior to July 2014, the host wrote a review about the guest, which became public immediately. Then the guest could read the review and decide if he or she wanted to leave a review about the accommodation.
Airbnb introduced two changes to make sure that these factors didn't negatively influence the integrity of the system. First, the company implemented a simultaneous review reveal process. This meant that neither hosts nor guests could view a review about themselves until they had written one.
And second, the company tackled the "discomfort" issue by offering a $25 Airbnb coupon to renters as an incentive for writing something--anything at all--about their stay.
Having taken a in-depth look at the review process, Airbnb had identified three main psychological factors that, previous to the changes, created an opening for dishonest reviews:
1. Fear of retaliation
The hosts wanted to avoid having guests respond to a bad review with a bad review of their own, so they left out the negative.
2. Induced reciprocity
In this case, the host left a false positive review of the renter, hoping that the guest would feel obligated to reciprocate with a positive review.
Uneasiness with criticism is one of the main reasons why both parties often would opt out of leaving a review altogether, but a little bribery, apparently, goes a long way toward getting more honesty.