评测编码代理在跨版本升级中的长周期开发能力。
RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
- 基于17个开源项目的真实版本升级，设计115个长周期开发任务。
- 平均修改3700行代码、涉及51个文件，测试多目标实现能力。
- 顶尖模型仅完成39.1%，凸显长周期软件开发仍属难题。
编码代理正越来越多地应用于实际软件开发中，而单次版本迭代通常需要数月的跨文件协同工作。然而，现有基准主要聚焦于从Python仓库中修复单一缺陷的任务，采用粗粒度的通过/失败评估，难以捕捉真实工程规模下的长周期、多目标开发。为填补这一空白，我们提出RoadmapBench，一个包含115个长周期编码任务的基准，源自17个开源仓库和5种编程语言的真实版本升级。每个任务将代理置于源版本代码快照，并提供多目标路线图指令，要求其实现目标版本引入的功能，平均需修改3,700行代码、覆盖51个文件。我们在十三个前沿模型上进行系统性评估，发现即使最强的Claude-Opus-4.7也仅能解决39.1%的任务，最弱模型仅为5.2%，远低于现有缺陷修复基准的表现，表明长周期软件开发仍是未解难题。
原文摘要 · Abstract (English)
Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。