用视觉因果数据预训练,让视觉语言模型回答更可信。
Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

- 构建包含视觉因果关系的FaithfulQA数据集,引导模型理解图像与问题关联
- 在6个VQA基准上验证,准确率提升且强化学习训练更稳定
- 适合需要高可信度视觉推理的应用,如医疗、自动驾驶
强化学习(RL)是提升视觉语言模型(VLM)推理能力的重要方法。然而,直接对多模态推理进行强化学习会导致不稳定性,原因在于模型过度依赖语言先验、忽视视觉证据,以及生成看似流畅但缺乏视觉依据的推理过程。为此,本文提出一种忠实预热策略(FWS):从六个通用VQA基准中筛选出具有明确视觉-语言因果关系的样本,构建FaithfulQA数据集,每个图像-问题对均包含视觉观察、问题要求、常识知识、领域知识及最终答案。随后使用基于VLM的判别器进一步净化数据,确保强因果一致性与视觉可信性。该预热阶段使模型在后续稀疏答案级奖励的强化学习优化前,具备理解因果驱动的视觉语言模式的能力。实验表明,这种忠实监督显著提升了答案准确率,稳定了强化学习训练,并减少了无视觉支持的推理。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。