Having a voice assistant device in your home, one driven by Alexa, Siri, Cortana, or the Google Assistant sure is handy. But which of these four devices is handiest? Which will do the best job of understanding your instructions and answering your questions?

Every year, researchers at the venture capital firm Loup Ventures look for the answers by challenging smart speaker devices with 800 different questions and commands in five categories. The idea is to determine which are most useful at everyday tasks, including everything from solving math problems to answering history questions to re-ordering paper towels to finding the nearest pizza place. 

The four devices they tested were a second generation Amazon Echo, a Google Home Mini, an Apple HomePod, and a Cortana-driven Harman Kardon Invoke smart speaker. In order to end up with an apples-to-apples comparison, and because using a smart speaker in the home is significantly different from using a mobile device, they tested smart speakers only, and didn't include, say, Siri on an iPhone in the mix. Here's what they learned:

1. Google Assistant is the smartest.

Google Assistant responded correctly to 88 percent of the questions and commands, compared with 75 percent for Siri, 73 percent for Alexa, and an embarrassing 63 percent for Cortana. That 88 percent showing for Google was mostly the result of its relatively poor performance in carrying out commands, where it scored only 73 percent. It scored 94 or 95 percent in the Information, Navigation (i.e. directions), and Local (where's the nearest pizza?) categories. Google led the pack in last year's bake-off as well, so no big surprises there.

2. They're all getting smarter fast.

But it is interesting to see just how much smarter these smart speakers are getting. For one thing, they understand questions and commands very, very well, with Google Assistant properly understanding 100 percent of the questions it was asked, and the others scoring 99 percent or better. "Nearly every misunderstood question involved a proper noun, often the name of a local town or restaurant," the researchers reported. "Both the voice recognition and natural language processing of digital assistants across the board has improved to the point where, within reason, they will understand everything you say to them."

Not only did they understand the questions and commands well, they are all getting better at providing the right responses. Google Assistant improved 7 points from 81 percent right last year. Cortana also improved 7 points and Alexa improved 9 points. But the HomePod, tested when in launched in February 2018, made a dramatic 23-point leap from 52 percent right nine months ago to 75 percent right in the most recent test, mainly because Apple enabled several new features in the months since launch.

Still, all smart speakers are getting smarter, which makes them more and more useful in real-world ways. The researchers note that all except HomePod can play a requested radio station, Google Assistant and Alexa can check the status of a flight, and all four can read a bedtime story.

3. Siri is great at following commands.

The only category Google lost is the command category, with Siri getting things right 85 percent of the time, compared with Google Assistant's 73 percent. Alexa and Cortana scored 68 percent and 65 percent, respectively. The researchers thought this was because the HomePod passes on most non-music queries to an associated mobile device, and Siri is very well integrated with calendar, messaging and other functions. At the same time, they thought that because the HomePod specializes in music, it was also better able to follow music-related commands.

4. Google Assistant is a better shopper than Alexa.

This is the one category where you might assume Alexa would outshine the others, built as it is by one of the world's biggest retailers. But no, Alexa's showing in this category was poor at 52 percent, beaten by Google at 86 percent and even Siri at 56 percent. (Cortana, at 31 percent, doesn't even seem to be trying in this category.)

Why did Alexa do so badly? Obviously, Alexa is well suited to ordering items on Amazon, but when it comes to doing anything else, it lags behind.

The researchers illustrated the problem with the query, "How much would a manicure cost?"

Alexa's response: "The top search result for manicure is Beurer Electric Manicure & Pedicure Kit. It's $59 on Amazon. Want to buy it?"

Google Assistant's response: "On average, a basic manicure will cost you about $20. However, special types of manicures like acrylic, gel, shellac, and no-chip range from about $20 to $50 in price, depending on the salon."

Based on available data, the researchers believe most people don't use their smart speakers to buy stuff most of the time. (In our household, that's the only way we ever buy toilet paper, paper towels, or coffee filters, but perhaps we're an exception.) But, they say, even for households like ours, Google Express, available through the Google Assistant, is just as useful as ordering on Amazon with Alexa.

Which one should you buy?

What should you do if you don't yet have a smart speaker and want to give one a try? Based on these results, Cortana isn't much of a contender (sorry, Microsoft). Apple's HomePod might make sense if you're deeply into the Apple ecosystem, and if you're willing to pay at least $250 for that deeper Apple integration and a superior speaker. But if a pretty good speaker is good enough, and you don't want to break the bank, that leaves the various flavors of Echo and Google Home. We've had an Echo since they first came out, and our Android phones come with the Google Assistant. So, though it may be an unfair comparison, I've informally and unintentionally tested them against each other many times when I've asked Alexa a question, gotten no useful response, then pulled out my smartphone or tablet, asked the Google Assistant the same question, and gotten the correct answer.

We have three Echo devices in our house, but I'm still thinking about getting an inexpensive Google Home Mini. I'll still ask Alexa to send toilet paper or play my tunes on Spotify. But, when I call out a question like "When did Charlemagne live?" or "How long will it take to get to Bothell?" It would be nice to get the right answer on the first try.

Published on: Dec 27, 2018
Like this column? Sign up to subscribe to email alerts and you'll never miss a post.