提出多模态语义扰动法，有效检测视觉语言模型的测试集泄漏问题。
Contamination Detection for VLMs using Multi-Modal Semantic Perturbation
- 通过可控的多模态语义扰动，检测模型是否依赖测试数据
- 在多种污染策略下，该方法均保持高检测准确率
- 适合关注模型真实性能的开发者与评估者使用
视觉语言模型（VLMs）在众多基准任务上表现优异，但其基于互联网规模、常为私有的预训练语料库，可能因测试集泄露导致性能虚高。尽管已有工作针对大模型提出去污染和基准重设计等缓解策略，但针对污染VLM的检测方法仍研究不足。本文故意污染开源VLM在主流基准上的表现，发现现有检测方法要么失效，要么行为不一致。为此，我们提出一种基于多模态语义扰动的简单而有效的检测方法，证明受污染模型在受控扰动下无法泛化。进一步在多种真实污染策略下验证，结果表明该方法具有强鲁棒性和有效性。代码与扰动数据集已公开于 https://github.com/jadenpark0/mm-perturb。
原文摘要 · Abstract (English)
Recent advances in Vision-Language Models (VLMs) have achieved state-of-the-art performance on numerous benchmark tasks. However, the use of internet-scale, often proprietary, pretraining corpora raises a critical concern for both practitioners and users: inflated performance due to test-set leakage. While prior works have proposed mitigation strategies such as decontamination of pretraining data and benchmark redesign for LLMs, the complementary direction of developing detection methods for contaminated VLMs remains underexplored. To address this gap, we deliberately contaminate open-source VLMs on popular benchmarks and show that existing detection approaches either fail outright or exhibit inconsistent behavior. We then propose a novel simple yet effective detection method based on multi-modal semantic perturbation, demonstrating that contaminated models fail to generalize under controlled perturbations. Finally, we validate our approach across multiple realistic contamination strategies, confirming its robustness and effectiveness. The code and perturbed dataset are released at https://github.com/jadenpark0/mm-perturb.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。