arXiv:2507.04943cs.CVcs.CL2025-07EMNLP被引 1

通过闭环训练让模型‘看两次、反向思考’,减少多模态模型的幻觉

ReLoop: "Seeing Twice and Thinking Backwards" via Closed-loop Training to Mitigate Hallucinations in Multimodal understanding

  • 设计环形结构,引入三重一致性反馈机制
  • 在多个基准上显著降低幻觉率,提升输出真实性
  • 适合关注多模态模型可靠性与事实一致性的研究者

尽管多模态大语言模型(MLLMs)在开放域视觉问答中取得显著进展,但仍易产生与输入语义矛盾或失真的幻觉输出,严重影响其可靠性和事实一致性。现有方法多依赖外部验证或事后修正,缺乏训练阶段直接验证输出的内在机制。为此,我们提出ReLoop,一种统一的闭环训练框架,通过“看两次、反向思考”促进跨模态理解的一致性。ReLoop采用环形结构,集成三个互补的一致性反馈机制:冻结的语义重构、视觉描述与注意力对齐模块,共同实现语义可逆性、视觉一致性与可解释注意力。实验表明,该方法在多个基准上有效降低幻觉率,为缓解多模态模型幻觉提供了稳健方案。代码与数据将在最终版本发布。

原文摘要 · Abstract (English)

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in open-ended visual question answering, they remain vulnerable to hallucinations. These are outputs that contradict or misrepresent input semantics, posing a critical challenge to the reliability and factual consistency. Existing methods often rely on external verification or post-hoc correction, lacking an internal mechanism to validate outputs directly during training. To bridge this gap, we propose ReLoop, a unified closed-loop training framework that encourages multimodal consistency for cross-modal understanding in MLLMs. ReLoop adopts a ring-shaped structure that integrates three complementary consistency feedback mechanisms, obliging MLLMs to "seeing twice and thinking backwards". Specifically, ReLoop employs the frozen Consistency Feedback Plugin (CFP), comprising semantic reconstruction, visual description, and an attention supervision module for attention alignment. These components collectively enforce semantic reversibility, visual consistency, and interpretable attention, enabling the model to correct its outputs during training. Extensive evaluations and analyses demonstrate the effectiveness of ReLoop in reducing hallucination rates across multiple benchmarks, establishing a robust method for hallucination mitigation in MLLMs. We will release our source code and data in the camera-ready version.

多模态幻觉抑制闭环训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。