arXiv:2506.02791cs.SEcs.AI2025-06被引 6

代码智能模型评估中,数据污染未必导致性能高估。

Contamination Means Overestimation? A Fine-Grained Empirical Study in Code Intelligence

  • 按代码任务分场景构建四类污染测试,细致分析影响。
  • 大模型直接推理时,成对污染显著提升性能;小模型则不明显。
  • 研究挑战了污染必然高估的旧观念,适合模型评估者参考。

近年来,代码智能在自动化软件工程领域愈发重要。随着预训练语言模型(PLMs)和大语言模型(LLMs)的广泛应用,数据污染及其对模型性能评估的影响引发关注。以往研究多聚焦样本级污染,忽视代码智能中普遍存在的部分污染情形。本文系统开展细粒度实证研究,考察主流代码任务中的数据污染影响。实验涵盖RoBERTa、GPT-2等PLMs,以及LLaMA、StarCoder等LLMs,覆盖代码翻译、生成、摘要三类任务,涉及Java与Python两种编程语言。根据代码实践将污染分为输入仅、输出仅、非配对、配对四种类型,并设置对照组进行验证。结果表明:在PLMs的预训练-微调-推理范式下,即使刻意引入配对污染,也未导致显著性能高估;但直接推理或小规模微调时污染效应显现。相比之下,采用预训练-推理范式的LLMs受配对污染显著影响。其余污染场景对两类模型均无显著影响。研究挑战了污染必然导致性能高估的普遍认知,为代码智能模型的评估与部署提供新视角。

原文摘要 · Abstract (English)

In recent years, code intelligence has gained increasing importance in the field of automated software engineering. Meanwhile, the widespread adoption of Pretrained Language Models (PLMs) and Large Language Models (LLMs) has raised concerns regarding data contamination and its potential impact on model performance evaluation. Previous studies mainly focused on sample-level contamination, ignoring partial contamination scenarios that are pervasive in code intelligence. This paper fills this gap and presents a systematic empirical study to investigate the fine-grained data contamination on mainstream code tasks. Our study involves diverse representative PLMs: RoBERTa and GPT-2, and LLMs: LLaMA and StarCoder, covering three major tasks: code translation, code generation, and code summarization, across two Programming Languages (PLs): Java and Python. We categorize contamination scenarios into four types according to the code intelligence practice, namely input-only, output-only, unpaired, and paired contamination settings, and construct corresponding experimental and control groups for exploration. Experimental results show that, under the pre-training, fine-tuning, and inference paradigm adopted by PLMs, even deliberately injecting paired contamination does not lead to significant performance overestimation. But direct inference or small-scale fine-tuning uncovers the contamination effects. In contrast, LLMs with pre-training and inference paradigm are significantly affected by the paired contamination. Apart from the above, other contamination scenarios have no impact on both PLMs and LLMs. Our findings challenge the conventional belief that contamination inevitably leads to performance overestimation, providing new insights into the evaluation and deployment of code intelligence models.

代码智能数据污染模型评估大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。