用强化学习在推理时修正视觉语言模型的幻觉问题。
Mitigating Image Captioning Hallucinations in Vision-Language Models
- 仅微调语言模型层归一化参数，避免重训练和额外模型。
- 在LLaVA和InstructBLIP上分别降低15.4%和17.3%幻觉率。
- 无需额外数据或模型，适合资源受限场景快速部署。
视觉语言模型（VLMs）中的幻觉问题影响其可靠性与实际应用，通常源于预训练数据与测试样本间的分布差异。现有方法如重新训练或微调需大量计算资源和人工数据收集，而集成方法则引入额外的VLM带来成本。为此，我们提出一种基于强化学习的测试时适应框架，在不重训练且无需辅助VLM的情况下缓解推理过程中的幻觉。通过仅更新语言模型层归一化中的可学习参数（约占模型参数的0.003%），该方法缩小了测试样本与预训练样本间的分布差距。同时设计基于CLIP的幻觉评估模型，为VLM提供双重奖励信号。实验表明，该方法在LLaVA和InstructBLIP上分别实现15.4%和17.3%的幻觉率下降，优于当前最佳基线68.3%的幻觉缓解效果，验证了其有效性。
原文摘要 · Abstract (English)
Hallucinations in vision-language models (VLMs) hinder reliability and real-world applicability, usually stemming from distribution shifts between pretraining data and test samples. Existing solutions, such as retraining or fine-tuning on additional data, demand significant computational resources and labor-intensive data collection, while ensemble-based methods incur additional costs by introducing auxiliary VLMs. To address these challenges, we propose a novel test-time adaptation framework using reinforcement learning to mitigate hallucinations during inference without retraining or any auxiliary VLMs. By updating only the learnable parameters in the layer normalization of the language model (approximately 0.003% of the model parameters), our method reduces distribution shifts between test samples and pretraining samples. A CLIP-based hallucination evaluation model is proposed to provide dual rewards to VLMs. Experimental results demonstrate a 15.4% and 17.3% reduction in hallucination rates on LLaVA and InstructBLIP, respectively. Our approach outperforms state-of-the-art baselines with a 68.3% improvement in hallucination mitigation, demonstrating its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。