构建真实项目迁移评估基准，检验大模型在代码适配中的可靠性。
TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
- 自动化构建含失效测试的项目迁移数据集
- 11个主流模型在验证集上表现受限，错误率超30%
- 适合关注代码生成可靠性的开发者与研究者
随着自动化软件工程的发展，研究重心逐渐转向反映工程师日常工作的实际任务。其中，软件迁移——将代码适配至不断演进的环境——是关键环节却长期被忽视。本文提出TimeMachine-bench，一个面向真实世界Python项目的迁移任务评估基准。该基准包含因依赖更新导致测试失败的GitHub仓库，构建过程全自动化，支持动态更新；同时人工验证筛选出可解问题子集以保障评估有效性。我们在该子集上评估了基于11个模型（涵盖强开源模型与前沿LLM）的代理基线系统，结果表明尽管大模型在迁移任务中展现出一定潜力，但仍面临严重可靠性挑战：包括利用低测试覆盖率产生的虚假解，以及因工具使用策略不佳导致的冗余修改。数据集与实现代码已公开于https://github.com/tohoku-nlp/timemachine-bench。
原文摘要 · Abstract (English)
With the advancement of automated software engineering, research focus is increasingly shifting toward practical tasks reflecting the day-to-day work of software engineers. Among these tasks, software migration, a critical process of adapting code to evolving environments, has been largely overlooked. In this study, we introduce TimeMachine-bench, a benchmark designed to evaluate software migration in real-world Python projects. Our benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates. The construction process is fully automated, enabling live updates of the benchmark. Furthermore, we curated a human-verified subset to ensure problem solvability. We evaluated agent-based baselines built on top of 11 models, including both strong open-weight and state-of-the-art LLMs on this verified subset. Our results indicated that, while LLMs show some promise for migration tasks, they continue to face substantial reliability challenges, including spurious solutions that exploit low test coverage and unnecessary edits stemming from suboptimal tool-use strategies. Our dataset and implementation are available at https://github.com/tohoku-nlp/timemachine-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。