arXiv:2609.04898cs.CLcs.AI2026-09

开源平台评估代码重构智能体,揭示有效策略与成本关系。

RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents

论文配图:RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents
图 1 · 摘自论文原文
  • 构建固定环境,系统测试模型、执行方式与提示设计对重构效果的影响。
  • 基于AST的分块比简单分块提升25%-30%成功率,检索增强优于无检索基线。
  • 适合研究重构智能体的开发者与评测人员,支持可复现、可审计的实验。

代码库级重构要求智能体在不改变程序行为的前提下,跨多个相互依赖文件传播单一修改,但目前尚无评估框架能独立分离影响代理性能的设计因素。本文提出RefactorPlatform,一个开源评估平台,固定环境并显式变化各设计维度:模型主干(通过OpenRouter和GitHub Copilot CLI)、执行模式(基线、检索增强、多智能体)及提示具体性。每次运行在隔离工作区中进行,包含实时终端流、任务级日志记录(令牌、差异、对话)、基于AST的验证以及可导出的遥测数据,支持审计与复现。在100个跨文件RefactorBench任务上,针对四个模型族进行测试,结果显示:基于AST的分块在所有提示模式下均比单纯按令牌窗口分块高出25%-30%;而简单的检索方法表现低于无检索基线;精简的检索增强单智能体(86%)优于我们测试的子智能体配置(66%),且在任务传递中,委托失败的案例未见于检索增强方案;检索带来的准确率提升足以抵消其令牌开销,使成功重构的单位成本保持不变。RefactorPlatform已开源,旨在实现重构智能体评估的可复现性与可审计性。

原文摘要 · Abstract (English)

Repository-scale refactoring requires coding agents to propagate a single change across many interdependent files without altering program behavior, yet to our knowledge no existing harness isolates the design choices that determine agent success on this task. We present RefactorPlatform, an open-source evaluation harness that holds the environment fixed and varies each design axis explicitly: model backbone (via OpenRouter and GitHub Copilot CLI), execution regime (baseline, retrieval-augmented, and multi-agent), and prompt specificity. Each run executes in an isolated workspace with live terminal streaming, per-task logging of tokens, diffs, and transcripts, AST-based verification, and exportable telemetry for audit and reproduction. Demonstrating the platform on 100 multi-file RefactorBench tasks across four model families, we illustrate the analyses it supports: AST-aware chunking outperforms naive token-window chunking by 25-30% across prompt modes, whereas naive retrieval falls below the retrieval-free baseline; a lean retrieval-augmented single agent (86%) beats the sub-agent configuration we evaluated (66%) on matched tasks with no task passing under delegation that fails under retrieval; and retrieval's accuracy gains absorb its token overhead, leaving cost per successful refactoring unchanged. RefactorPlatform is open-sourced to make refactoring-agent evaluation reproducible and auditable.

代码生成智能体评估重构开源平台

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。