通过细粒度视觉对比优化，有效减少多模态幻觉。
Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization
- 在多图上下文中进行视觉对比，确保目标函数数学严谨。
- 在5个基准上表现最优，显著降低幻觉率。
- 适合关注模型可靠性与视觉推理的开发者与研究者。
多模态幻觉仍是视觉语言模型（VLMs）的持续挑战。标准文本直接偏好优化（DPO）因缺乏显式视觉监督而难以缓解该问题。现有方法虽引入视觉偏好DPO，通过对比原图与负样本，但其目标函数存在理论不一致问题，且依赖粗粒度负样本，易导致捷径学习。本文提出上下文视觉对比优化（IC-VCO），通过将对比图像置于共享多图上下文中，确保数学上的严格性。进一步提出视觉对比蒸馏（VCDist），一种辅助的可靠性门控正则化器，促进多图对比训练与单图推理间的一致性。最后提出一种对比样本编辑策略，通过精确语义扰动生成难例负样本。在五个基准上的实验表明，IC-VCO整体性能最佳，且样本编辑策略效果显著。代码与数据已开源。
原文摘要 · Abstract (English)
Multimodal hallucination remains a persistent challenge for Vision-Language Models (VLMs). Standard textual Direct Preference Optimization (DPO) often fails to mitigate it due to a lack of explicit visual supervision. While existing works introduce visual preference DPO by contrasting original images against negative ones, they suffer from a theoretically inconsistent objective caused by partition function mismatches and rely on coarse-grained negatives that could enable shortcut learning. In this work, we propose In-Context Visual Contrastive Optimization (IC-VCO). By placing contrastive images within a shared multi-image context, IC-VCO ensures a mathematically rigorous objective. We further introduce Visual Contrast Distillation (VCDist), an auxiliary reliability-gated regularizer that encourages consistency between multi-image contrastive training and single-image inference. Finally, we propose a contrastive sample editing strategy that generates hard negatives via precise semantic perturbations. Experiments on five benchmarks demonstrate IC-VCO's best overall performance and the effectiveness of our sample editing strategy. Code and data are available at https://github.com/OPPO-Mente-Lab/IC-VCO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。