Book Excerpt: Inside Companies’ High-Payoff Generative AI Experiments

Inc contributor Peter Cohan has been digging deep into how companies are using generative AI products. His new book, Brain Rush: How to Invest and Compete in the Real World of Generative AI (2024) published by Apress, looks at the different ways businesses are trying to benefit from this technology, and where they face challenges. An excerpt: 

By early 2024, academics and consultants highlighted positive outcomes from early experiments with generative AI. Indeed a 2023 study by economic researchers found ChatGPT and similar tools resulted in “more productive workers, happier customers, and higher employee retention.” In September 2023, a consulting firm executive highlighted significant economic payoffs from deploying generative AI in ways that boosted employee productivity and increased revenue considerably – providing companies a high return on the consulting fees expended to plan and implement these generative AI applications.

Economic researchers found compelling evidence of success in a study of early generative AI experiments. A 2023 National Bureau of Economic Research working paper presented the results of “nearly 5,200 customer support agents at a Fortune 500 software firm who gained access to a generative AI-based assistant in a phased rollout between November 2020 and February 2021.” The generative AI tool made real-time recommendations to the support agents on how to respond to customer inquiries and provided links to company documents. Compared to a control group, workers assisted by the chatbot resolved 14% more issues per hour, completed conversations faster, and were a bit more successful in resolving problems. The chatbot boosted the productivity of the least experienced workers by 35%. 

Erik Brynjolfsson, a senior fellow at the Stanford Institute for Economic Policy Research, said this productivity boost was much higher than the 1%-2% productivity gains companies typically obtain from new information technology. Brynjolfsson and his colleagues found these productivity gains sprang from the chatbot’s identification – “by digesting millions of transcripts of service interactions” – and dissemination of tactics used by the most successful agents. Customers were happier because their problems were resolved more quickly, and support agents were less likely to resign. “We don’t know for sure why this occurred, but I would guess that it’s more enjoyable to be in a job where the customers like you and you can solve customer problems faster,” Brynjolfsson said. His conclusion was companies would benefit from using generative AI to augment “high-skilled workers so the system can continue to learn from them.”

Consultants also expressed optimism about generative AI’s benefits for workers. One industry expert, Michele Goetz, Vice President and Principal Analyst at Forrester Research, told me clients were paying millions in consulting fees for help deploying generative AI in their companies. The reason? Through lower costs and/or higher revenues, clients expected generative AI applications to earn back those consulting fees – which could range from $500,000 to a range from $10 million to $50 million – within three months of deployment. Forrester saw clients getting their money back in well under a year. “Companies forecast a payback within three months of deployment,” she told me. “In some cases, companies see results in two to eight weeks. In other cases, the time to value is three to six months.” Companies achieve this payback through quantitative and qualitative improvements. “One company reduced its call center costs by $80 million in the course of a year,” Goetz explained. “The benefits included cost savings from shifting responses to level 0 and level 1 questions from human agents to generative AI virtual assistance and agents using the technology to respond faster and more accurately to customer inquiries. In addition, generative AI increases customer satisfaction and improves net promoter scores.” 

Generative AI also added to company revenue. For example, the technology enabled a cruise ship company to boost its revenue by $1 billion by recommending ancillary activities that their customers activated during the voyage. An Asian bank’s [generative AI-powered] virtual investment assistance delivered an “additional $500 million worth of trading volume,” she told me. In 2023, early adopters of generative AI in industries such as construction, travel, retail, health care, and energy achieved higher productivity and changed customer behavior in the wake of their generative AI experiments. These companies were also evaluating whether the high cost and limitations of these applications – such as the need to hire specialized talent, and manage legal and privacy risks – exceeded their benefits. 

Here are two examples:

• Wayfair’s AI-powered interior decorator. Wayfair, the Boston-based online furniture retailer, introduced Decorify, a free generative AI tool launched in July 2023 to help customers redesign their living rooms. Fiona Tan, Wayfair’s chief technology officer, said customers upload a photo of their living rooms to Decorify, which created “photorealistic images” of proposed designs and prompted consumers with real products similar to the ones in the photo. Decorify could help consumers shop for items “that are really hard for you to articulate what it is you really want. Being able to see that is helpful,” Tan said.
• Expedia’s personalized travel assistant. Seattle-based Expedia’s chatbot enabled travelers to ask for recommendations and make bookings to help them “explore and discover,” Expedia CEO Peter Kern said. Expedia viewed the experiment as a way to get consumers comfortable with generative AI – while maintaining the company’s current booking process. Expedia also used AI to automate customer-servicecall summaries to reduce costs. By providing “tailored” booking help, Expedia aimed to keep travelers from “fishing around in the dark,” Kern said. Expedia’s chief technology officer Rathi Murthy said the company had overhauled its huge store of customer data so it could train ChatGPT – providing customers with greater personalization.

In early 2024, these experiments left many questions unanswered:

• How many customers were finding these generative AI assistants helpful?
• If so, which aspects were most useful? If not, was there any reason to continue the experiment?
• Did these assistants generate clear payoffs – through lower costs, higher productivity, or increased revenue?
• Were these applications free of compliance, cybersecurity, copyright, or other sources of risk?
• Did the experiments point the way to enhancements that would boost the value of these assistants?

Excerpted from Peter Cohan’s new book Brain Rush: How to Invest and Compete in the Real World of Generative AI (2024) published by Apress.