arXiv:2511.03182cs.SEcs.LG2025-11

测试代码大模型编辑方法在API更新下的鲁棒性,发现多数方法效果差且会破坏原有性能。

Understanding Robustness of Model Editing in Code LLMs

  • 构建包含2040个问题的可控基准,模拟140种合成API修改
  • 单次编辑后泛化能力弱,多数成功是绕过而非真正迁移API
  • 连续编辑导致性能崩溃,编译阶段错误占失败主因

用于代码的大语言模型(LLMs)在软件开发中日益重要,但其在预训练后保持静态,而API和软件库持续演进。模型编辑提供了一种轻量级替代方案,可在不重新训练的情况下集成API更新。然而,现有编辑方法是否能正确实现API迁移、泛化到未见任务,以及保持对未修改API任务的性能仍不清楚。本文构建了一个受控基准,基于HumanEval、MBPP和APPS数据集,包含2,040个问题及140种合成的API修改,并配备执行沙箱以确保在标准Python语义下使用已编辑的API。评估了多种先进编辑方法在三种代码LLM上的表现,采用基于执行的指标区分真正的API采纳与基于绕行的任务完成。单次编辑下,编辑模型在未见任务中的泛化能力差,许多看似成功的案例实为绕行而非真实迁移;涉及未修改API的任务性能也下降,其中基于记忆的方法和微调比定位-编辑类方法更能保持特异性。在连续编辑下,大多数方法-模型组合在泛化和特异性上均接近零的Pass@k,表明存在显著的非目标干扰。两因子Shapley分解显示,单次编辑泛化失败主要源于编译阶段,而特异性失败多发生在编译后;连续编辑时,失败则主要由编译驱动。

原文摘要 · Abstract (English)

Large language models (LLMs) for code are increasingly used in software development, but they remain static after pretraining while APIs and software libraries continue to evolve. Model editing offers a lightweight alternative to retraining for incorporating API updates, yet it remains unclear whether existing editing methods can induce correct API migration, generalize that behavior to unseen tasks, and preserve performance on tasks involving unmodified APIs. We present a controlled benchmark for evaluating model editing under API updates in code LLMs, built from HumanEval, MBPP, and APPS, with 2,040 problems spanning 140 unique synthetic API modifications, together with an execution sandbox that enforces edited APIs under standard Python semantics. We evaluate several state-of-the-art editing methods on three code LLMs under both single-edit and successive-edit regimes using execution-based metrics that distinguish successful API adoption from workaround-based task completion. Under single edits, edited models generalize poorly to unseen uses of the modified API, and many apparent successes are workaround-based rather than true API migrations. Performance on tasks involving unmodified APIs also degrades, although memory-based methods and fine-tuning preserve specificity better than locate-then-edit methods. Under successive edits, most method-model combinations collapse to near-zero Pass@k on both generalization and specificity, revealing substantial interference beyond the target edits. A two-factor Shapley decomposition further shows that single-edit failures on generalization include a substantial compilation component, whereas specificity failures are more often post-compilation. Under successive edits, failures become predominantly compilation-driven.

模型编辑代码生成鲁棒性API更新

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。