针对多模态假信息检测,提出自适应多维度思考框架,提升推理准确性和泛化能力。
MMD-Thinker: Adaptive Multi-Dimensional Thinking for Multimodal Misinformation Detection
- 设计专用思维模式,结合指令微调注入通用大模型
- 在8000+图文对数据集上实现领先性能,跨域检测效果优异
- 适合需要高可信度假信息检测的平台与研究者使用
多模态假信息在社交媒体中泛滥,尤其在AI生成内容(AIGC)时代持续演化。现有基于通用多模态大语言模型(MLLMs)的方法虽取得显著进展,但仍存在两大局限:一是推理能力不足,通用模型采用统一推理范式,缺乏任务特定知识,导致解释和判断不准确;二是推理偏见,单一思维模式难以应对快速演变、复杂的多模态假信息。为此,本文提出MMD-Thinker,一种两阶段自适应多维度思维框架。首先,为多模态假信息检测定制专属思维模式;其次,通过任务特定指令微调将该模式注入通用MLLMs;第三,采用混合优势函数的强化学习策略,激励有效推理轨迹。此外,构建了包含8000余张图文对的多模态假信息推理(MMR)数据集,涵盖推理过程与分类标签。实验表明,MMD-Thinker在多个域内与域外基准数据集上均达到最先进水平,同时具备灵活推理与高效资源利用能力。代码将开源于GitHub。
原文摘要 · Abstract (English)
Multimodal misinformation floods on various social media, and continues to evolve in the era of AI-generated content (AIGC). The emerged misinformation with low creation cost and high deception poses significant threats to society. While recent studies leverage general-purpose multimodal large language models (MLLMs) to achieve remarkable results in detection, they encounter two critical limitations: (1) Insufficient reasoning, where general-purpose MLLMs often follow the uniform reasoning paradigm but generate inaccurate explanations and judgments, due to the lack of the task-specific knowledge of multimodal misinformation detection. (2) Reasoning biases, where a single thinking mode make detectors a suboptimal path for judgment, struggling to keep pace with the fast-growing and intricate multimodal misinformation. In this paper, we propose MMD-Thinker, a two-stage framework for multimodal misinformation detection through adaptive multi-dimensional thinking. First, we develop tailor-designed thinking mode for multimodal misinformation detection. Second, we adopt task-specific instruction tuning to inject the tailored thinking mode into general-purpose MLLMs. Third, we further leverage reinforcement learning strategy with a mixed advantage function, which incentivizes the reasoning capabilities in trajectories. Furthermore, we construct the multimodal misinformation reasoning (MMR) dataset, encompasses more than 8K image-text pairs with both reasoning processes and classification labels, to make progress in the relam of multimodal misinformation detection. Experimental results demonstrate that our proposed MMD-Thinker achieves state-of-the-art performance on both in-domain and out-of-domain benchmark datasets, while maintaining flexible inference and token usage. Code will be publicly available at Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。