测试大模型能否像人类一样优雅重构代码，发现其识别人类重构选择能力较弱。
CodeTaste: Can LLMs Generate Human-Level Code Refactorings?
- 构建CodeTaste基准，基于真实开源项目重构数据。
- 模型能实现细节明确的重构，但难发现人类实际选择的优化方向。
- 提出先提议后实现框架，提升与人类重构习惯对齐度。
LLM编码代理虽能生成可运行代码，但常引入复杂性、重复和架构债务。人类开发者通过重构（保持行为不变的结构优化）解决此类问题。本文研究代理是否（i）能可靠执行重构，以及（ii）能否识别真实代码库中人类开发者实际选择的重构方案。为此，我们构建了CodeTaste基准，数据源自大型多文件开源项目中的真实重构。评估采用仓库测试套件验证功能正确性，并结合定制化静态检查，利用数据流分析判断是否移除了不良模式并引入了期望模式。结果表明：当重构指令详细时，代理表现良好；但在给定修改区域时，难以发现人类实际采用的重构方案。提出‘提议-实现’分解策略可提升对齐度，且在实现前选择最匹配的提议能进一步增益。CodeTaste为评估代理与人类重构决策一致性提供基准，并可能作为偏好信号。代码与排行榜已公开。
原文摘要 · Abstract (English)
LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. We investigate whether agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. To this end, we construct CodeTaste, a benchmark mined from large multi-file open-source refactorings. To score solutions, we combine repository test suites that measure functional correctness with tailored static checks that verify removal of undesired and introduction of desired code patterns using dataflow reasoning. Our results show a clear gap: agents perform well at implementing refactorings that are specified in detail, but often fail to discover the human refactoring choices when given a focus area for changes. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases. We release the benchmark, leaderboard, and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。