构建首个全流程代码修复与生成评估基准，揭示智能编码助手的短板。
A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents
- 引入中间步骤真实标签，追踪需求理解、规划到编码全过程
- 31.5%平均解决率，隐含需求理解是最大瓶颈（占比24.5%-46.0%）
- 适用于评估和改进代码生成智能体，尤其关注需求推理能力
大型语言模型驱动的编码助手在修改代码库时面临挑战，现有基准仅以最终补丁是否通过测试来评价，无法反映失败轨迹的具体偏差。为此，我们提出SWE-RPG，一个基于仓库级别的基准，结合可执行补丁评估与（1）需求澄清和（2）实现规划的验证真值（GT）。这些中间真值支持对编码代理整个流程（从需求理解、规划、编码到提交）进行回溯性诊断。SWE-RPG包含来自31个Python和Java仓库的163个任务，涵盖113个缺陷修复和50个功能新增。我们评估了包括Claude Code、Codex和OpenCode在内的3个编码代理，使用6个大语言模型后端（如Claude-Sonnet-5和GPT-5.6-Terra）。结果显示，当前主流编码代理在实际仓库中仍难以完成用户请求，平均解决率为31.5%。中间真值诊断进一步发现，隐含需求恢复是主要瓶颈，占代理运行的24.5%至46.0%。这表明隐含需求理解是提升编码代理的关键方向。基准数据与评估代码已公开于https://github.com/Xin-Zhou-smu/SWE-RPG-Bench。
原文摘要 · Abstract (English)
Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository-level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at https://github.com/Xin-Zhou-smu/SWE-RPG-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。