(This is a sidebar to Jakob Nielsen’s article AI May Improve Employee Productivity by 66%. It contains a detailed description of the first of the three research studies analyzed in that article.)

Erik Brynjolfsson and colleagues from Stanford University and MIT conducted the first of the three studies discussed in the main article. Brynjolfsson is the world’s leading expert on the economic impact of technology use.

The study participants were 5,000 customer-support agents working for a large enterprise-software company. They were provided with an AI tool built on the large language models developed by OpenAI. (Further details are not provided in the paper.) The ecological validity of this study is enhanced by the fact that the agents were studied in their natural environment while performing their everyday work. (In contrast, studies 2 and 3 employed artificial settings where study participants performed assigned tasks instead of their usual work.)

It was possible to measure the impact of the AI tool because it was rolled out gradually to various agents. Thus, at any given time, some agents used the AI tool, whereas others continued to work in the traditional way, without AI.

Support agents who used the AI tool could handle 13.8% more customer inquiries per hour. This difference is statistically significant at the p < 0.01 level. Furthermore, work quality improved by 1.3%, as measured by the share of customer problems that were successfully resolved. This slight increase was only marginally significant at p = 0.1. (A second measure of quality, the net-promoter score reported by the customers, was not significantly different between the two conditions. The one thing I make of this finding is that customers could not tell whether the agent employed AI to solve their problem.)

The study also found that AI assistance benefited the less-skilled and less-experienced agents the most. The lowest-performing 20% of the agents (the bottom quintile) improved their task throughput by 35% — two and a half times as much as the average agent. In contrast, the best-performing 20% of the agents (top quintile) improved their task throughput by only a few percent, and possibly not at all (because the margin of error included the zero point, which means that the improvement could have been zero).

One possible explanation of the fact that the best agents didn’t improve with the use of AI comes from a more detailed analysis of the support sessions. This analysis found that productivity increased primarily when an agent followed the recommendations from the AI, as opposed to relying on their own personal ideas. Unfortunately, the most-experienced agents were the least likely to follow the AI’s recommendations — probably because they were highly confident in their own insights. In other words, experience backfired because the best agents were more likely to dismiss the AI, even when it was right.

The AI tool also accelerated the learning curve for new agents. On average, new agents complete 2.0 customer inquiries per hour. After 8 months, their productivity of increases to 2.5 inquiries per hour (without AI help). In contrast, the agents who started using the AI tool right off the bat reached this level of performance in only two months. Even better, the AI-using agents reached a performance of 3.0 inquiries per hour after 5 months of experience, meaning that they had already become substantially better than the non-AI users with 8 months of experience.

Reference

Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond (2023): Generative AI at Work. National Bureau of Economic Research working paper 31161. https://www.nber.org/papers/w31161