(This is a sidebar to Jakob Nielsen’s article AI May Improve Employee Productivity by 66%. It contains a detailed description of the third of the 3 research studies analyzed in that article.)

Sida Peng and colleagues from Microsoft Research, GitHub, and MIT conducted the last of the three studies discussed in the main article. In this study, 70 programmers were asked to implement an HTTP server in JavaScript. Half were in a treatment group and used the GitHub Copilot AI tool, whereas the others formed the control group that didn’t use any AI tools. The participants had an average coding experience of 6 years and reported spending an average of 9 hours per day coding. So, it would be reasonable to consider the participants to be experienced developers.

The main finding was that programmers who did not use AI completed the task in 160.89 minutes (2.7 hours) on average, whereas the programmers who had AI assistance completed the job in 71.17 minutes (1.2 hours). The difference between the two groups was statistically significant at the level of p = 0.0017. 

This means that, without AI, in a 40-hour work week, programmers with no AI help would be able to implement 14.9 problems like the sample task, whereas programmers with Copilot help would be able to implement 33.7 such problems. In other words, productivity in the form of task throughput increased by 126% for developers who used the AI tool.

The difference in success (i.e., the ability to complete the assignment) between the two groups was not statistically significant, so I won’t discuss it further.

Compared with study 2, study 3 has two advantages:

  • Participants were a more advanced type of business professional than the mainstream office workers in study 2.
  • The task was more substantial: with no AI, it took almost 6 times longer to perform than the task used in study 2 (writing a business document). A classic weakness of most research is that it studies small tasks that are not representative of realistic business problems. I still won’t say that a programming assignment that can be completed in less than 3 hours is a true challenge for professional developers, but at least building an HTTP server is not a toy problem.

The study participants’ subjective estimate was that using the AI tool decreased their task time by 35% relative to the time to code the server without the tool. This corresponds to a guessed increase in productivity of 54% more coding problems in a week. 

As we know from the data, the actual productivity improvement was 126%, so why did the participants think they had experienced only a 54% productivity gain? Well, as always, we can’t rely on what people say or what they guess would happen with a different user interface than the one they used. We must watch what actually happens. Only observational data is valid. Guessed data is just that: guesses.

Detailed data analysis shows that programmers with fewer years of experience benefited more from the AI tool, though the effect is only marginally significant, at p = 0.06. Also, programmers who spent fewer hours per day coding benefited more from the AI tool than participants who coded for more hours per day. This second effect is significant, at p = 0.02.

Taken together, these two findings suggest that less-skilled programmers benefit somewhat more from AI than more-skilled programmers do. (To be precise: the study didn’t truly measure programming skills, but it’s reasonable to assume that people with more years of experience and more hours of daily practice will indeed be better, on average, than people with less experience.)

A downside of study 3 is that the researchers did not assess the code quality produced by the two groups. This would certainly be possible to do, both with objective metrics (say, how efficiently each implementation runs in terms of computer resources consumed) and estimated metrics (say, how easy the code would be to maintain, as estimated by a few expert programmers). Of course, I can (and will!) always ask for more from any research study. No research is perfect. As with studies 1 and 2, I applaud Peng and colleagues for their work.

Reference

Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer (2023): The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. Available at Arxiv: https://arxiv.org/abs/2302.06590 or https://doi.org/10.48550/arXiv.2302.06590