用在线数据优化视觉语言模型，有效减少幻觉现象。
Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
- 基于在线数据设计动态重加权的直接偏好优化算法
- 在多个基准上将幻觉率降低超79%，13B模型超越GPT-4V
- 通过二值化标注确保训练数据纯净，防止引入新幻觉
大型视觉语言模型（LVLMs）在多模态任务中表现优异，但幻觉问题仍具挑战性。本文分析发现，在线数据显著优于离线数据，亟需高效可靠的偏好标注方法。现有标注方式本身引入幻觉，可能加剧模型幻觉。为此，提出训练一个二值化幻觉分类器，确保后续对齐使用的样本无幻觉。为进一步挖掘在线数据潜力，设计一种鲁棒的迭代直接偏好优化（DPO）算法，采用动态样本重加权机制。在三个基准上与8个先进基线对比，实验显示该方法使LLaVA-1.5-7B在MMHalBench上的幻觉率降低50.8%，在Object HalBench上的平均幻觉率降低79.5%；更显著的是，该方法充分释放开源模型潜力，使LLaVA-1.5-13B性能超越GPT-4V。
原文摘要 · Abstract (English)
Recently, large vision-language models (LVLMs) have risen to be a promising approach for multimodal tasks. However, principled hallucination mitigation remains a critical challenge.In this work, we first analyze the data generation process in LVLM hallucination mitigation and affirm that on-policy data significantly outperforms off-policy data, which thus calls for efficient and reliable preference annotation of on-policy data. We then point out that, existing annotation methods introduce additional hallucination in training samples, which may enhance the model's hallucination patterns, to address this problem, we propose training a hallucination classifier giving binary annotations, which guarantee clean chosen samples for the subsequent alignment. To further harness of the power of on-policy data, we design a robust iterative direct preference optimization (DPO) algorithm adopting a dynamic sample reweighting scheme. We conduct comprehensive experiments on three benchmarks with comparison to 8 state-of-the-art baselines. In particular, our approach reduces the hallucination rate of LLaVA-1.5-7B on MMHalBench by 50.8% and the average hallucination rate on Object HalBench by 79.5%; more significantly, our method fully taps into the potential of open-source models, enabling LLaVA-1.5-13B to even surpass the performance of GPT-4V.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。