研究指令文件如何影响AI生成代码的合并成功率。
Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests

- 分析148个项目的指令文件对AI生成代码的影响。
- 27.7%项目合并率提升超20%,但近四分之一反而下降。
- 高质量指令需结构清晰、篇幅较长,适合工程化管理。
AI代理(如GitHub Copilot)在软件工程任务中以协作方式参与代码生成,通过拉取请求(Agentic-PRs)提出修改。为提升效率,开发者常编写指令文件,指导代理如何导航项目、定位组件、运行测试并遵循最佳实践。本文基于AIDev数据集中的15,549个Agentic-PRs,分析148个项目的指令文件前后表现。结果发现:指令文件并非必然提升效果——27.7%项目合并率提升至少20%,但26.35%反而下降;代码变更量与合并耗时、评论数等指标也呈现类似波动。初步观察显示,合并率提升的项目通常拥有更长、结构更优的指令文件,包含更多章节与子节。研究呼吁将指令文件开发视为一项可工程化的活动,即“Instructions-as-Code”。
原文摘要 · Abstract (English)
AI-agents (e.g., GitHub Copilot) collaborate as teammates in different software engineering tasks, including code generation proposed through pull requests (Agentic-PRs). For better agent efficiency, developers create instruction files that guide the AI-agents, including how to navigate the project, locate the right components, run tests, respect best practices, and more. In this paper, we investigate the relationship between the creation of these instructions and the performance of AI-agents in creating better pull requests, which have a higher chance of success (i.e., the merge rate), address more complex tasks (e.g., code churn), and require less effort to be merged (e.g., time to merge). To this end, we analyze 15,549 agentic PRs from 148 projects in the AIDev dataset. Using the three dimensions, we compare each project before and after the creation of the instruction files. We find that specifying instructions for AI-agents does not necessarily lead to better results. With the instruction files, 27.7\% of the projects increased their merge rate by at least 20\%, while 26.35\% decreased it. The same observation is seen with the amount of changes (e.g., code churn, number of modified files) and with the efforts to merge an agentic PR (e.g., merge time and number of comments). From a first exploration, we find that projects that managed to increase their merge rate have substantially longer instruction files, which are also well structured into a higher number of sections and sub-sections. Our results motivate the need for research to assist practitioners in framing the development of instruction files as a software engineering activity (aka, \textbf{Instructions-as-Code}).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。