It was probably inevitable that Ambarish Mitra would start an augmented reality company like Blippar, which lets you look at real-world objects enhanced with text and digital graphics through your smartphone camera. After all, Mitra seems to have lived his entire life in an augmented reality of his own, one that reads like a mash-up of Charles Dickens, Gabriel García Márquez, and Salman Rushdie.
Just listen to Mitra's tale of how he got his start as an entrepreneur. In 1997, he was a 17-year-old runaway in the slums of New Delhi, sleeping in a shack with mud-and-cow-dung walls and making his living peddling tea and magazines, when he came across an advertisement for a business-plan contest run by an Indian tech company. First prize: $10,000. A feminist movement was then sweeping India, and Mitra had the idea of providing free internet access to women who couldn't afford to pay for it. He wrote up his pitch and mailed it off--then returned to his life as a chai vendor, confident that was that. How could it not be? Even if the judges liked his plan, what were the chances they would be able to find him in his mud shack, with no telephone or mailing address?
Except they did. A postman following the instructions he'd written out in lieu of a return address left a letter with the local tobacconist, informing Mitra that he'd been shortlisted and inviting him to pitch in person. He did, and won. Three years later, WomenInfoLine, the company born of that idea, was sold in a deal that made him a wealthy man--and vindicated his decision to leave home and a life of predictable middle-class comfort. (If this sounds suspiciously close to the plot of the movie Slumdog Millionaire, well, the parallels have frequently been noted in press reports. Mitra's sister and a friend from that era confirm parts of his account, though documentation on the contest is thin.)
"That post shouldn't even have come back to me," Mitra marvels, at Blippar's San Francisco headquarters, 20 years and two continents later. A serene, round-faced man of 37, he wears his usual uniform: T-shirt, gemstone rings on each finger of his right hand, and a flat-billed baseball cap emblazoned with "Spiritual Gangster." The rings are custom made; he never goes out without them. The cap, one of several, is purchased from the yoga-wear company Spiritual Gangster, but Mitra has adopted its brand name as his personal mantra. The phrase captures the two sides of his personality: the fighter and the philosopher. "Every human has dichotomies," he says. "Most of us have two fields of view in life."
The seen and the unseen: That's the dichotomy Mitra has spent the past six years seeking to bridge with Blippar. Along the way, he has raised nearly $100 million and enlisted 300 employees, aiming to build a visual search engine that can recognize anything you point your phone at and serve up relevant facts, games, shopping information, and user-generated content. Think Pokémon Go meets Wikipedia meets Foursquare. Already, 65 million people have downloaded the app to "blipp" an ad or product, generating tens of millions of dollars in revenue from Blippar's advertising partners.
And the company has only just begun marketing directly to consumers, potentially opening up hundreds of millions of dollars more in revenue. With Pokémon Go and Snapchat's face filters turning the masses on to augmented reality, Mitra claims Blippar's moment has arrived. "We've taken the majority of human knowledge and created a visual DNA for it," he says.
That's the ambition, anyway. But even if the age of AR is upon us, Blippar is hardly alone in heralding it--and there's a whiff of magical thinking to Mitra's belief that his startup, which is still losing tens of millions of dollars per year and struggling with technical glitches, will be its biggest winner. The tech industry's largest players are all racing ahead with their own efforts to define what the augmented future will look like, and capture the profits from it. Microsoft has the HoloLens, a mixed reality headset designed primarily for gaming, while Facebook has its own Oculus Rift headset. Meanwhile, Google and its parent company, Alphabet, have made one stab after another at VR and AR, including a secretive mixed reality startup called Magic Leap. The few who have seen Magic Leap in action report it to be truly mind-blowing; meanwhile, Blippar's app sometimes has trouble distinguishing a coffeepot from a teakettle.
But where others see big threats, Mitra recognizes only tiny wrinkles. The glitches in how his app works are temporary byproducts of a powerful deep-learning process, he says, one that's well on the way to creating "a superhuman brain" with "the entire knowledge repository of the whole world." He also argues that the company's head start in weaving together disparate disciplines like 3-D rendering and computer vision will give competitors no choice but to use its operating system. "We started this well before any of the big guys even thought about it," he says.
Mitra's vision is itself a form of augmented reality: what he is sure of will be superimposed on what is. One thing he's not imagining is the scope of what he's undertaken. "Making the physical world digital is a bit of a holy grail in the tech community," says Brian Blau, vice president of research at tech advisory firm Gartner. "That's really what they're after, but almost all these other augmented reality businesses are after it as well."
Still, Blau believes Blippar is looking at a gigantic opportunity--if it can survive as an independent company until society is ready to embrace its augmented future. "There's a lot of possibility there," he says, "but also a lot of risk."
Mitra's particular blend of toughness and ethereality has its roots in Dhanbad, in eastern India, where he grew up. Historically, the city was a center of wisdom, home to one of the world's first universities. Then the Industrial Revolution happened, and the vast coal fields underneath Dhanbad became more valuable than any of the learning happening there. You can guess what followed. "Spiritualism went down and corruption went up," Mitra says. "Literally, the dark side of human nature came out by flipping the very land. This has been core to my foundation."
When his family moved from Kolkata, where Mitra was born, to Dhanbad, it was one of India's most violent towns. Mitra's father, Aghore, was an engineer for the steel company Tata, and the family lived in a secure, company-run enclave, but at every opportunity Mitra and his friends played in abandoned coal mines, running along ropeways over open pits. "When I look back, the stuff we did, every day someone could have died," he says. "In rural parts of India, adventure is built in--so when you get into cities, you find all parts of life not a problem."
Mitra would soon put that to the test. As the eldest son of a Tata engineer, he was guaranteed a coveted job at the company after graduation--but he was bored by studying and failed his year-end exams two years in a row. His family had by then moved to Delhi; one day, ashamed of being held back, Mitra ran away to the slums, in New Delhi. A note to his parents said he was in Mumbai. "He was a risk-taker, very adventurous," says Shamim Ahmed, who shared a mud-walled room with Mitra and seven others, and who runs a school in the northern city of Gaya. "He wanted to do something big in his life."
When his internet company got acquired in 2000, Mitra suddenly found himself charged with a sense of invincibility. He moved to England and resumed his studies, while doing contract IT work for the government and working at a series of startups. The first one he joined quickly fizzled. Then he launched a company of his own, SwapShop, a sort of Craigslist for people looking to trade or give away unwanted items. To code the site, he enlisted his best friend's younger brother, a Web developer named Omar Tayeb. The startup sputtered out, but the partnership endured: Mitra and Tayeb tried again with what would eventually become Stuck, an advice-oriented social network. It never got off the ground, and Mitra's savings finally ran out. He moved into a tiny apartment and took a job as head of innovation with an insurance company called Swiftcover, which had recently been acquired by the insurance giant AXA. Tayeb got a job there too, and they were able to launch Stuck in-house.
By then, the two were close friends; Tayeb was no longer just the kid who sat in the back seat on his brother's outings. One night in 2010, the partners were blowing off steam at a pub when Mitra paid for drinks with a 20-pound note. When the bartender was slow to make change, Mitra mused about how cool it would be if the Queen's face on his bill could come to life, like the magical portraits in Harry Potter. Wait a second--could they make that really happen?
They could. Tayeb managed to code a crude version within a few weeks; it was enough to persuade both friends to quit their jobs and try starting yet another company.
If Tayeb is Blippar's original engineer, Mitra is its visionary. He has his own version of the "reality distortion field" famously attributed to Steve Jobs. Like the Apple co-founder, he's not so much a technologist as an idea man and a brilliant communicator who can tell a story in a way that gets others excited to be part of it. "He somehow has a weird confidence that everything is going to work out," and it's contagious, Tayeb says.
That's exactly how Mitra charmed his way into Blippar's first, extravagant offices. He was determined to set up shop in London's posh Covent Garden-Holborn neighborhood to help win big advertising accounts. "It was completely unaffordable for us as a company with zero revenue and no funding," Tayeb says. "Trust me," Mitra said, and proceeded to sweet-talk the CEO of an outdoor advertising company into giving Blippar free office space for more than a year, convincing him that augmented reality would make billboards so much more valuable. By the time Blippar had to pay its own way, it was able to afford offices 150 feet down the street.
Not everything was so cheap, of course. Mitra and Tayeb quickly realized they needed to hire expensive talent to build the powerful image-recognition engine they envisioned. They could do it without raising a lot of money if they could sign up some big advertising customers that would pay them to create augmented reality content. Despite having no real users to speak of, Mitra sold the concept to brands like Tesco and Cadbury, and signed them up for five-figure national campaigns.
That was enough business to get them noticed by bigger investors. Tayeb had built the first version of Blippar's app using software licensed by Qualcomm, where executives soon saw a spike in traffic to one tiny customer. Jason Ball, who makes tech investments for Qualcomm's venture arm, asked for a meeting, at which Mitra's salesmanship once again worked. He explained that brands put the Blippar logo on their packaging and in ads, encouraging their customers to download the app--all at no cost to Blippar.
Ball was astonished. "I was like, 'So you're not marketing your product, you're not spending anything on user acquisition, and the customer is paying you for the pleasure of doing this on your behalf?' " he says. "Dude, how fast can I give you a check?"
By 2014, Mitra and Tayeb had proved Blippar was viable, but it was in danger of becoming a glorified digital ad agency. It sold exactly the sort of product big marketers were looking for: something immersive that could hold consumers' attention in a world where online ad blockers were making them ever harder to reach. That was good for Blippar's revenue, but bad for customer retention: Since the company relied on marketers to tell people to download the app, Blippar's employees spent their time making content for these advertisers. The app functioned almost like a white-label product with no real user base of its own.
But Mitra and Tayeb had an idea how to build one. They knew that after people blipped ads or products, they often pointed their cameras at other things around them to see what content appeared--only to be disappointed when nothing popped up. What they needed was to make everything blippable, so users could treat Blippar as a camera-powered search engine or a visual Wikipedia. If you're curious about anything--a car, a cucumber, Vladimir Putin, the Mona Lisa--you should be able to point your phone's camera at it and find out more. And not just what it is, but how it relates to everything else. If it's a dog, what breed? If it's a blouse, where can it be purchased, and what skirts or pants might pair well with it? "We want to recognize everything in the physical world and give people content on top of it," as Mitra puts it.
So the founders focused their energies on the new mission. In February 2015, they raised more money to hire dozens of engineers and open up big new offices in San Francisco and nearby Mountain View. Tayeb moved from London to Mountain View that summer, making it the new center of operations, and Mitra relocated to San Francisco.
The move was crucial to Blippar's new strategy. Swimming in the Silicon Valley talent pool made Mitra and Tayeb realize how naive they'd been about their task. Until one fateful recruiting dinner, they'd thought they could scrape so much of the internet that their app would be able to do a reverse-image search on anything the camera saw. "We were going to build a database of a trillion, trillion images," Mitra says.
Then an engineer they were courting, named Xuejun Wang, told Mitra point blank there was no information architecture in the world that could make such a database manageable. Even if the company could build it, it would be so slow as to be useless, taking at least 25 seconds of processing time to return a result. What Blippar needed instead, Wang said, was a so-called deep-learning system. It would mimic the human brain, which processes images by understanding their shared attributes: Humans don't have to learn to recognize every type of chair that exists; they need to understand only what makes something a chair.
Wang eventually agreed to join Blippar as vice president of infrastructure. For the past 18 months, the company's engineers have been working to help the app understand what it sees by creating "stencils," which allow its machine-learning algorithms to group things into categories. Through this approach, Mitra claims, Blippar now recognizes almost 75 percent of the objects in the physical world, and more for certain categories, including dog breeds and cars. "Even the human eye cannot distinguish between the 2016 and 2017 Prius," Mitra brags, "but Blippar can."
After my first meeting with Mitra, I took Blippar for a spin around San Francisco. That quickly highlighted both the product's strengths and its shortcomings, which have earned it an average rating of 2½ stars in Apple's App Store. Blippar was quick to recognize a box of Kraft macaroni and cheese, but it failed to identify a grapefruit or a pair of scissors. When I pointed the camera at an object it couldn't identify, the screen filled up with word associations, like "produce" and "healthful."
In our next interview, I told Mitra Blippar wasn't as smart as I'd expected. Yes, he said, but that's because it's working to become even smarter. The word cloud is there to show that the app is guessing and learning in the same way humans do. "If I bring you into an unfamiliar church and ask you where we are, you won't be able to name the church--but you'll be able to describe it," he says.
By the time we were done, Mitra had me doing exactly what he does: seeing Blippar for what it could be, not just what it is now.
There's a lot that Blippar still needs to learn, though. Last year, it sought some buzz with a new product, one that would instantly recognize faces. Blippar signed deals with 25 celebrities, and booked a day of television interviews and events around the world to publicize the app's new facial-recognition feature. The idea was to encourage users to upload their own faces and create profiles for themselves, giving Blippar a social layer that would make it more attractive to advertisers. It was the first time the company was spending significant dollars on consumer marketing, and it was going to be huge.
And then, one night in December, just hours before the new version of the app was to be published, Mitra called off the launch. The update was glitchy, with frequent crashes and a just-noticeable delay in response time. It wasn't the sort of thing that would impress new users.
"It was a hard call," Mitra says two days later. "But we did the right thing."
Blippar won't be able to afford many such stumbles--especially given the scope and breadth of its competition. Google is on its second iteration of a similar product, now called Tango, that lets Android users do things like take augmented museum tours and view the night sky with the names of the planets superimposed on their screens. And Google is strong exactly where Blippar is vulnerable: on engineering.
"It's a lot for a startup to do," says Qi Lu, a mentor to Mitra who headed up search at Microsoft and is now COO of the Chinese search giant Baidu. Lu says Mitra and Tayeb have hit on a big idea with a large potential market and a strong business model--but he's not sure they can compete with companies that have thousands of engineers working on machine learning and image recognition.
Competition isn't necessarily a bad thing. With a private valuation of $1 billion, Blippar makes what Lu calls an attractive and affordable target for companies like Google, Apple, and Microsoft. In fact, the founders say acquisition offers have already been on the table, including one worth $1.5 billion.
For a startup, that's still a long way from turning a profit--Blippar reported a $31.3 million loss for the 16 months ending March 31, 2016--that's a lot of money to walk away from. But Mitra says he's not interested in a soft landing. And Tayeb points out that Blippar's singular focus on one type of technology may help it break through, as huge Silicon Valley companies get distracted. "There's a reason Google didn't come out with Facebook and Facebook didn't come out with Snapchat," he says. "This can't be just a side project in a bigger company."
Those companies also all grew up in an era when text, not images, was the dominant form of information. That leaves an opportunity to be the first successful image-native tech company, Mitra says, pointing out that three-sevenths of the world's population is either illiterate or minimally literate. Who's going to be their Google?
"We are heading toward a world, inside five years, where computer vision, A.I., and natural-language translation will make everyday information about the stuff in front of you absolutely seamless," Mitra says. "We want to play a big role in shaping that world."
*CORRECTION: The original version of this story mischaracterized Mitra's exit from WomenInfoLine. It did not conduct an initial public offering of its own but was acquired by the publicly-traded Mudra Consultants, which then changed the name to M.WomenInfoLine. The original text also misstated the timing of Axa's acquisition of Swiftcover and failed to note that Stuck was launched while Ambarish Mitra and Omar Tayeb were employees of Swiftcover, although Mitra says most of the work on the product was done before they were hired there.