测试大模型对代码库版本演化的理解能力，发现其普遍缺乏版本感知。
LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
- 构建多版本库的评测基准LibEvoBench，评估模型对演进API的理解。
- 主流模型在版本演进的API上性能下降，稳定API则无差异，表现版本盲。
- 指定目标版本无效，但引入相关文档可显著提升准确率，适合关注代码生成可信度的研究者。
大型软件项目常依赖旧版库，而库的API随版本持续演进，这给大模型带来挑战：需掌握多个版本的API知识，而非仅最新或常见版本。当前大模型基于时间混合的语料训练，缺乏显式版本推理机制，导致调用错误版本的API。为系统评估此现象，我们提出LibEvoBench，一个覆盖多个常用Python库版本的多任务基准，并引入软件演化理解得分（SEUS）衡量模型在演进API上的表现一致性。结果表明，主流模型普遍版本盲：面对演进的API性能下降，而稳定API则跨版本表现一致。仅指定目标版本无改善作用，但提供相关文档可显著提升准确性。这些发现揭示了现有训练范式的系统性局限，推动面向时间锚定知识的新方法发展。
原文摘要 · Abstract (English)
Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one. However, current LLMs are trained on temporally mixed corpora and lack explicit mechanisms for such version-specific reasoning, leading to anachronistic errors - calling APIs as they exist in a different library version. To systematically evaluate this phenomenon, we introduce LibEvoBench, a multi-task benchmark spanning multiple versions of widely used Python libraries, along with a new metric, the Software Evolution Understanding Score (SEUS), to measure models' consistency when working with evolving APIs. Our results show that state-of-the-art models are largely version-oblivious: performance degrades for evolving APIs, while for stable APIs it remains the same across versions. Moreover, simply specifying the target version provides no benefit, while relevant documentation significantly boosts models' accuracy. These findings highlight a systematic limitation of current training paradigms and motivate new approaches for temporally grounded knowledge in code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。