代码大模型更倾向输出训练中见过的错误代码,而非修复版本。
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
- 通过数据归属测试识别模型是否见过错误或修复代码,评估其偏好。
- 67%的代码案例中模型未接触过任何版本,但见过错误代码时更易复现错误。
- 尽管评分指标显示模型偏好正确修复,但实际生成仍易复制记忆中的错误。
大型语言模型在代码生成与调试中广泛应用,但其输出仍可能含错误,源于训练数据。区分模型是偏好正确代码,还是熟悉但错误的版本,可能受训练过程中暴露内容影响。本文提出一种暴露感知评估框架,量化模型对错误代码与修复代码的偏好如何受训练数据暴露影响。基于ManySStuBs4J基准,利用Data Portraits对Stack-V2语料库进行成员资格检测,估算每个错误与修复版本是否在训练中出现。随后按暴露情况分组,采用代码补全和多种基于似然的评分指标比较模型偏好。结果发现,67%的样本中两类代码均未出现在训练数据中;当仅有一方存在时,修复版本更常被训练数据包含。在模型生成中,模型复现错误代码的频率远高于修复版本,且在接触过错误代码的情况下这一倾向加剧,而接触过修复版本仅带来微弱改善。在似然评分中,最小与最大词元概率指标始终偏好修复代码,显示出稳定的正确性倾向;而基尼系数等指标在仅见过错误版本时则反转偏好。结果表明,暴露情况会扭曲对模型错误修复能力的评估,提示模型可能在实践中传播记忆中的错误。
原文摘要 · Abstract (English)
Large language models are increasingly used for code generation and debugging, but their outputs can still contain bugs, that originate from training data. Distinguishing whether an LLM prefers correct code, or a familiar incorrect version might be influenced by what it's been exposed to during training. We introduce an exposure-aware evaluation framework that quantifies how prior exposure to buggy versus fixed code influences a model's preference. Using the ManySStuBs4J benchmark, we apply Data Portraits for membership testing on the Stack-V2 corpus to estimate whether each buggy and fixed variant was seen during training. We then stratify examples by exposure and compare model preference using code completion as well as multiple likelihood-based scoring metrics We find that most examples (67%) have neither variant in the training data, and when only one is present, fixes are more frequently present than bugs. In model generations, models reproduce buggy lines far more often than fixes, with bug-exposed examples amplifying this tendency and fix-exposed examples showing only marginal improvement. In likelihood scoring, minimum and maximum token-probability metrics consistently prefer the fixed code across all conditions, indicating a stable bias toward correct fixes. In contrast, metrics like the Gini coefficient reverse preference when only the buggy variant was seen. Our results indicate that exposure can skew bug-fix evaluations and highlight the risk that LLMs may propagate memorised errors in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。