实测发现2025年AI工具反而让资深开发者编码变慢19%
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- 通过随机对照试验，对比开发者使用与不使用前沿AI工具的效率差异
- 实际完成时间延长19%，远超开发者预估的20%节省，也违背专家预测
- 结果挑战主流认知，适合关注AI对真实开发影响的研究者和实践者
尽管广泛采用，但人工智能工具在真实软件开发中的影响仍缺乏研究。我们开展一项随机对照试验（RCT），评估2025年初前沿AI工具对有经验的开源开发者生产力的影响。16名具备中等AI使用经验的开发者，在平均有5年经验的成熟项目上完成了246个任务。每个任务被随机分配为允许或禁止使用2025年初的AI工具。允许使用时，开发者主要使用Cursor Pro代码编辑器及Claude 3.5/3.7 Sonnet模型。任务开始前，开发者预计使用AI可缩短24%的完成时间；完成后估算为20%。然而，实际结果显示，允许使用AI反而使完成时间增加19%——AI工具导致开发速度下降。这一结果与经济学专家（预测缩短39%）和机器学习专家（预测缩短38%）的预期相悖。为理解该现象，我们收集并评估了20项潜在影响因素，如项目规模、质量标准或开发者过往的AI使用经验。尽管无法完全排除实验设计干扰，但减速效应在多种分析中保持稳健，表明其不太可能仅由实验设计所致。
原文摘要 · Abstract (English)
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。