构建动态基准测试，评估大模型在Rust API演进中的适应能力。
RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation
- 自动化生成588个真实场景下的API演变任务
- 模型对行为变更类任务成功率仅38.0%
- 适合关注代码生成鲁棒性与版本兼容性的研究者
大型语言模型（LLMs）已成为软件开发中代码自动生成的关键工具。然而，面对Rust等快速演进语言的频繁应用编程接口（API）变更，这些模型在生成版本感知代码时面临挑战，导致兼容性问题和正确性错误。现有基准缺乏对模型应对API过渡能力的系统评估，依赖人工标注且版本信息有限。为此，我们提出RustEvo框架，可自动合成588个API变更（其中380个来自Rust标准库，208个来自15个第三方库），构建反映真实开发挑战的编程任务，涵盖四大演进类型：稳定化、签名变更、行为变更与弃用，其分布符合实际生态。实验表明，主流模型在稳定化API上平均成功率达65.8%，但在行为变更任务中仅38.0%，凸显模型对无签名变化的语义迁移识别困难。知识截止日期显著影响性能：截止前的API得分56.1%，截止后的任务仅为32.5%。检索增强生成（RAG）可平均提升13.5%成功率。研究强调了演进感知基准对提升模型在快速迭代软件生态中适应性的必要性。框架与数据集已公开于https://github.com/SYSUSELab/RustEvo。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become pivotal tools for automating code generation in software development. However, these models face significant challenges in producing version-aware code for rapidly evolving languages like Rust, where frequent Application Programming Interfaces (API) changes across versions lead to compatibility issues and correctness errors. Existing benchmarks lack systematic evaluation of how models navigate API transitions, relying on labor-intensive manual curation and offering limited version-specific insights. To address this gap, we present RustEvo, a novel framework for constructing dynamic benchmarks that evaluate the ability of LLMs to adapt to evolving Rust APIs. RustEvo automates dataset creation by synthesizing 588 API changes (380 from Rust standard libraries, 208 from 15 third-party crates) into programming tasks mirroring real-world challenges. These tasks cover four API evolution categories: Stabilizations, Signature Changes, Behavioral Changes, and Deprecations, reflecting their actual distribution in the Rust ecosystem. Experiments on state-of-the-art (SOTA) LLMs reveal significant performance variations: models achieve a 65.8% average success rate on stabilized APIs but only 38.0% on behavioral changes, highlighting difficulties in detecting semantic shifts without signature alterations. Knowledge cutoff dates strongly influence performance, with models scoring 56.1% on before-cutoff APIs versus 32.5% on after-cutoff tasks. Retrieval-Augmented Generation (RAG) mitigates this gap, improving success rates by 13.5% on average for APIs released after model training. Our findings underscore the necessity of our evolution-aware benchmarks to advance the adaptability of LLMs in fast-paced software ecosystems. The framework and the benchmarks are publicly released at https://github.com/SYSUSELab/RustEvo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。