通过一致性增强检测大模型数据污染，区分微调与泄露。
CAP: Data Contamination Detection via Consistency Amplification
- 利用模型输出一致性构建性能比，识别数据泄露。
- 在7个大模型和4个专业基准上验证，发现混合数据集易含污染。
- 无需额外条件，支持白盒黑盒，适用于多种评测场景。
大型语言模型广泛应用，但数据污染问题威胁其评估可靠性。现有检测方法多依赖特定任务或额外前提，实用性受限。本文提出一致性增强型数据污染检测框架CAP，引入性能一致性比率（PCR）通过语言模型的一致性来衡量数据集泄露。据我们所知，这是首个明确区分微调与数据污染的方法，对领域专用模型的污染检测至关重要。CAP适用于多种评测基准，且兼容白盒与黑盒模型。我们在7个LLM和4个领域专用基准上验证其有效性，结果表明来自多个数据源的复合基准尤其容易发生无意污染。代码将不久后公开。
原文摘要 · Abstract (English)
Large language models (LLMs) are widely used, but concerns about data contamination challenge the reliability of LLM evaluations. Existing contamination detection methods are often task-specific or require extra prerequisites, limiting practicality. We propose a novel framework, Consistency Amplification-based Data Contamination Detection (CAP), which introduces the Performance Consistency Ratio (PCR) to measure dataset leakage by leveraging LM consistency. To the best of our knowledge, this is the first method to explicitly differentiate between fine-tuning and contamination, which is crucial for detecting contamination in domain-specific models. Additionally, CAP is applicable to various benchmarks and works for both white-box and black-box models. We validate CAP's effectiveness through experiments on seven LLMs and four domain-specific benchmarks. Our findings also show that composite benchmarks from various dataset sources are particularly prone to unintentional contamination. Codes will be publicly available soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。