用多模型知识蒸馏，让小模型低成本高效识别被篡改的新闻图片。
Multi-MLLM Knowledge Distillation for Out-of-Context News Detection
- 用多个大模型生成预测和推理过程作为知识源。
- 仅用不到10%标注数据就达到顶尖性能。
- 适合资源有限但需高精度检测虚假新闻的场景。
多模态上下文外新闻是一种误导性信息，其中图像被脱离原始语境使用。现有方法多依赖多模态大语言模型（MLLM）进行检测，但小型MLLM在零样本下的表现有限，通常需大量标注数据或昂贵的GPT API调用进行微调，难以在低资源场景应用。为此，我们提出一种更标签高效、成本更低的方法：首先通过提示多个教师型MLLM生成标签预测及其对应推理理由，共同构成教师知识；随后引入两阶段知识蒸馏框架将该知识迁移至学生型MLLM。第一阶段使用全部训练数据对学生模型进行LoRA微调；第二阶段在教师预测冲突的数据点上，结合LoRA与DPO进一步微调学生模型，以挖掘复杂案例中的细微模式。实验表明，本方法仅使用不足10%的标注数据即可实现当前最优性能。
原文摘要 · Abstract (English)
Multimodal out-of-context news is a type of misinformation in which the image is used outside of its original context. Many existing works have leveraged multimodal large language models (MLLMs) for detecting out-of-context news. However, observing the limited zero-shot performance of smaller MLLMs, they generally require label-rich fine-tuning and/or expensive API calls to GPT models to improve the performance, which is impractical in low-resource scenarios. In contrast, we aim to improve the performance of small MLLMs in a more label-efficient and cost-effective manner. To this end, we first prompt multiple teacher MLLMs to generate both label predictions and corresponding rationales, which collectively serve as the teachers' knowledge. We then introduce a two-stage knowledge distillation framework to transfer this knowledge to a student MLLM. In Stage 1, we apply LoRA fine-tuning to the student model using all training data. In Stage 2, we further fine-tune the student model using both LoRA fine-tuning and DPO on the data points where teachers' predictions conflict. This two-stage strategy reduces annotation costs and helps the student model uncover subtle patterns in more challenging cases. Experimental results demonstrate that our approach achieves state-of-the-art performance using less than 10% labeled data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。