The last thing you want to do is give the users of your online service a reason to pick up the phone and call you. This is exactly the problem that startup darling Airbnb experienced in its early years, just at a time it was trying to scale

At one point Riley Newman, Airbnb head of data science, took a look the company's booking numbers and compared it to the number of customer service inqueries. "People were contacting us at a rate that was like one to one," Newman said Monday at DataBeat in San Francisco. About 500 people attended the event, which was hosted by VentureBeat.

"So here we thought we were building this really efficient platform that people would be able to use at scale," Newman said. "But really, from this perspective, we were like a phone-based booking service."

It was easy to see the larger issue. Airbnb's product team was interested in rapidly turning out cool new features, and the customer support team had its hands full trying to address all of the arising issues. Meanwhile, customer calls were generating lots of data about what needed fixing, and no one was acting on it. 

Now Newman needed to know what was happening on a more granular level so he could decide how to improve. 

Interestingly, he resolved not to let technology get in the way of the data. Newman said that some data science teams might be tempted to build something involved, like a natural language processing system, to sort through the hundreds of customer support tickets. 

But Newman was after an even quicker way to get to the root of Airbnb's problems, so he tasked humans with sorting through the tickets. "They'd go off, read them, and they'd come back and give us a presentation about what specifically we needed to do in order to mitigate this issue," Newman said. 

Some of the resulting solutions were quite small, and some were even commonsense, Newman said. For example, a few of the fixes by the product team included an FAQ page, a map showing where bookings were located, and a way to get in touch with the host. 

"All pretty straightforward, but these weren't things that we were prioritizing on our product roadmap previously," Newman said. He added that these quick and simple solutions helped to bring the customer contact rate down by 75 percent in a just a short amount of time. 

"A huge win for the scalability of the company, but also for the user experience that we aspire to," he said.