构建代码重构评测集，揭示语言模型在状态追踪上的关键缺陷
RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code
- 设计100个跨文件重构任务，需理解多文件依赖与指令约束
- 现有模型仅解决22%任务，人类开发者在短时内达87%正确率
- 引入状态感知机制可提升43.9%，适合研究智能编程代理的学者
语言模型代理与函数调用的进步使自主、反馈驱动的系统能在多个数字领域解决问题。为深入理解语言模型代理的独特局限，我们提出RefactorBench，一个包含100个精心设计的多文件重构任务的基准测试集，来自热门开源仓库。完成这些任务需全面探索多文件间的依赖关系，并严格遵循相关指令。每个任务由3条不同具体程度的自然语言指令定义，且相互独立，支持在同一仓库上构建更长的组合任务。基准测试显示，当前语言模型代理在基础指令下仅能解决22%的任务，而有时间限制的人类开发者可达87%。轨迹分析揭示了多种独特失败模式，尤其在追踪过往动作方面表现不佳。通过让基线代理基于状态表示进行条件判断，任务解决率提升了43.9%。我们进一步将该状态感知方法扩展至整个数字环境，并指出未来研究方向。RefactorBench旨在通过真实世界的多跳代码任务，推动语言模型代理的研究。
原文摘要 · Abstract (English)
Recent advances in language model (LM) agents and function calling have enabled autonomous, feedback-driven systems to solve problems across various digital domains. To better understand the unique limitations of LM agents, we introduce RefactorBench, a benchmark consisting of 100 large handcrafted multi-file refactoring tasks in popular open-source repositories. Solving tasks within RefactorBench requires thorough exploration of dependencies across multiple files and strong adherence to relevant instructions. Every task is defined by 3 natural language instructions of varying specificity and is mutually exclusive, allowing for the creation of longer combined tasks on the same repository. Baselines on RefactorBench reveal that current LM agents struggle with simple compositional tasks, solving only 22% of tasks with base instructions, in contrast to a human developer with short time constraints solving 87%. Through trajectory analysis, we identify various unique failure modes of LM agents, and further explore the failure mode of tracking past actions. By adapting a baseline agent to condition on representations of state, we achieve a 43.9% improvement in solving RefactorBench tasks. We further extend our state-aware approach to encompass entire digital environments and outline potential directions for future research. RefactorBench aims to support the study of LM agents by providing a set of real-world, multi-hop tasks within the realm of code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。