首次揭示大模型生成Verilog代码存在数据泄露风险，影响评估公正性。
VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination
- 用污染检测方法分析主流Verilog生成评测框架
- 多款主流大模型均存在数据泄露，影响评估结果可信度
- 适合关注AI代码生成可靠性与评测公平性的工程师
大型语言模型（LLMs）在代码生成方面取得显著进展，但在基准测试中存在数据污染问题——即评测数据意外泄露至预训练或微调数据集，导致评估结果失真。尽管该问题已广为人知，限制了其在工业界的采纳，但硬件编码领域却鲜有研究。本文首次针对Verilog代码生成的SOTA评测框架（VerilogEval和RTLLM），采用成熟的污染检测方法（CCD和Min-K% Prob），对包括CodeGen2.5、Minitron 4b、Mistral 7b、phi-4 mini、LLaMA-{1,2,3.1}、GPT-{2,3.5,4o}、Deepseek-Coder和CodeQwen 1.5在内的商业与开源模型进行分析，并涵盖基线与微调版本（RTLCoder和Verigen）。研究证实数据污染是严重问题，同时探索缓解策略及其对代码质量与评测公平性之间的权衡。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have revolutionized code generation, achieving exceptional results on various established benchmarking frameworks. However, concerns about data contamination - where benchmark data inadvertently leaks into pre-training or fine-tuning datasets - raise questions about the validity of these evaluations. While this issue is known, limiting the industrial adoption of LLM-driven software engineering, hardware coding has received little to no attention regarding these risks. For the first time, we analyze state-of-the-art (SOTA) evaluation frameworks for Verilog code generation (VerilogEval and RTLLM), using established methods for contamination detection (CCD and Min-K% Prob). We cover SOTA commercial and open-source LLMs (CodeGen2.5, Minitron 4b, Mistral 7b, phi-4 mini, LLaMA-{1,2,3.1}, GPT-{2,3.5,4o}, Deepseek-Coder, and CodeQwen 1.5), in baseline and fine-tuned models (RTLCoder and Verigen). Our study confirms that data contamination is a critical concern. We explore mitigations and the resulting trade-offs for code quality vs fairness (i.e., reducing contamination toward unbiased benchmarking).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。