评测大模型在代码迁移任务中的表现，发现其平均准确率仅26.5%。
CODEMENV: Benchmarking Large Language Models on Code Migration
- 构建专用基准测试CODEMENV，涵盖922个跨环境代码迁移案例
- GPT-4O表现最佳，通过率43.84%，但整体平均仅26.5%
- 模型更擅长新版本函数迁移，易误判无关逻辑变更
大型语言模型（LLMs）在多种软件工程任务中表现出色，但在代码迁移——即适配代码以在不同环境中运行——方面的效果仍缺乏系统研究。本文提出CODEMENV：跨环境代码迁移基准，包含922个来自19个Python和Java包的实例，覆盖三大核心任务：(1) 识别与特定版本不兼容的函数，(2) 检测函数定义的变化，(3) 将代码适配至目标环境。在七种LLMs上进行评估，平均pass@1率为26.50%，其中GPT-4O表现最优，达43.84%。关键发现包括：(i) LLMs对新版本函数更熟练，有助于老旧代码迁移；(ii) 模型有时会因逻辑不一致而错误识别与迁移环境无关的函数变更。数据集已公开于https://github.com/xdshen-ai/Benchmark-of-Code-Migration。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable capabilities across various software engineering tasks; however, their effectiveness in code migration, adapting code to run in different environments, remains insufficiently studied. In this work, we introduce CODEMENV: Code Migration Across Environment, a new benchmark specifically designed to assess LLMs' abilities in code migration scenarios. CODEMENV consists of 922 examples spanning 19 Python and Java packages, and covers three core tasks: (1) identifying functions incompatible with specific versions, (2) detecting changes in function definitions, and (3) adapting code to target environments. Experimental evaluation with seven LLMs on CODEMENV yields an average pass@1 rate of 26.50%, with GPT-4O achieving the highest score at 43.84%. Key findings include: (i) LLMs tend to be more proficient with newer function versions, which aids in migrating legacy code, and (ii) LLMs sometimes exhibit logical inconsistencies by identifying function changes irrelevant to the intended migration environment. The datasets are available at https://github.com/xdshen-ai/Benchmark-of-Code-Migration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。