As we do new things, we always want to make sure we're doing things that help the business and not hurt. We may have a feeling of intuition that something may be successful, but how do you really know?
We run experiments. For example, maybe I believe that emailing our users about how fun our games are will get them to train with the games more. So, first, I set a benchmark. I'll measure the average training time with our current user emails. Then, I'll split the users into three buckets: One group will get the originals, one will get the fun games email, and the rest will get an email that references how the games are backed by pretty hardcore neuroscience. Because we have 40 million users, we can run that experiment and get valid data about which is best in just a few months. (As it turns out, users prefer the science emails.)
This makes our decision process really transparent. It creates a merit-based culture. What we decide is not about title or rank. I can't just say we don't do something because I don't like it.
If you can run one small experiment every week, something tiny--changing a button on the purchase page, say--and it makes the user experience 1 percent better? Over the course of the year, you'll improve by 70 percent. That's what leads companies to double and triple sales every year.
As told to Inc. senior writer Burt Helm.