通过三招降低视觉大模型幻觉,提升推理准确性。
Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization
- 先生成精准描述再推理,避免错误前提传递
- 优先探索高奖励方差样本,增强多样性
- 用信息熵损失调节样本相似性,稳定训练
尽管多模态大语言模型(MLLMs)在各类任务中取得显著进展,其实际应用仍受幻觉问题严重制约,尤其在强化学习(RL)优化过程中更为突出。本文系统分析了RL训练下MLLM幻觉的根源,识别出三个关键因素:(1) 过度依赖链式视觉推理,初始描述不准确或冗余信息会锚定后续推断至错误前提;(2) 策略优化中探索多样性不足,导致模型生成过于自信但错误的输出;(3) 训练样本间存在破坏性冲突,神经切线核(NTK)相似性引发虚假关联与参数更新不稳定。为此,我们提出包含三个核心模块的综合框架:首先,在推理前引入规划与描述阶段,采用基于质量的描述奖励确保初始锚定准确;其次,根据奖励分布的均值与方差对样本分类,优先选择高方差样本以聚焦于多样且有信息量的数据;最后,通过分组样本对并施加InfoNCE损失,调节NTK相似性,推动相似样本分离、异质样本靠近,引导梯度交互保持平衡。实验表明,该方法显著降低幻觉率,并有效提升MLLM的推理准确性。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse tasks, their practical deployment is severely hindered by hallucination issues, which become particularly acute during Reinforcement Learning (RL) optimization. This paper systematically analyzes the root causes of hallucinations in MLLMs under RL training, identifying three critical factors: (1) an over-reliance on chained visual reasoning, where inaccurate initial descriptions or redundant information anchor subsequent inferences to incorrect premises; (2) insufficient exploration diversity during policy optimization, leading the model to generate overly confident but erroneous outputs; and (3) destructive conflicts between training samples, where Neural Tangent Kernel (NTK) similarity causes false associations and unstable parameter updates. To address these challenges, we propose a comprehensive framework comprising three core modules. First, we enhance visual localization by introducing dedicated planning and captioning stages before the reasoning phase, employing a quality-based caption reward to ensure accurate initial anchoring. Second, to improve exploration, we categorize samples based on the mean and variance of their reward distributions, prioritizing samples with high variance to focus the model on diverse and informative data. Finally, to mitigate sample interference, we regulate NTK similarity by grouping sample pairs and applying an InfoNCE loss to push overly similar pairs apart and pull dissimilar ones closer, thereby guiding gradient interactions toward a balanced range. Experimental results demonstrate that our proposed method significantly reduces hallucination rates and effectively enhances the inference accuracy of MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。