系统分析多模态大模型数据泄露问题，揭示训练中隐匿的隐私风险。
Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM
- 提出双模态与跨模态污染分类框架，量化评估多模态模型数据泄露
- 12个模型在5个基准上均现显著污染，部分源于单模态预训练阶段
- 为模型评测提供可信依据，适合关注AI安全与评估公平性的研究者
多模态大语言模型（MLLM）性能快速提升，但训练中无意记忆测试数据（即数据污染）严重影响评估公平性。现有针对单模态大模型的检测方法难以应对多模态数据复杂性及多阶段训练。本文提出分析框架MM-Detect，定义两类污染：单模态与跨模态，并在多项选择和图像描述类视觉问答任务中有效量化污染程度。对12个MLLM和5个基准的评估显示，污染现象普遍，尤其在专有模型和旧基准中更为严重。关键发现：污染有时源自单模态预训练阶段，而非仅来自多模态微调。该研究深化了对污染机制的理解，有助于改进评估实践，提升多模态模型可靠性。
原文摘要 · Abstract (English)
The rapid advancement of multimodal large language models (MLLMs) has significantly enhanced performance across benchmarks. However, data contamination-unintentional memorization of benchmark data during model training-poses critical challenges for fair evaluation. Existing detection methods for unimodal large language models (LLMs) are inadequate for MLLMs due to multimodal data complexity and multi-phase training. We systematically analyze multimodal data contamination using our analytical framework, MM-Detect, which defines two contamination categories-unimodal and cross-modal-and effectively quantifies contamination severity across multiple-choice and caption-based Visual Question Answering tasks. Evaluations on twelve MLLMs and five benchmarks reveal significant contamination, particularly in proprietary models and older benchmarks. Crucially, contamination sometimes originates during unimodal pre-training rather than solely from multimodal fine-tuning. Our insights refine contamination understanding, guiding evaluation practices and improving multimodal model reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。