通过分解推理过程,让视觉语言模型更依赖真实视觉信息。
Self-Rewarding Vision-Language Model via Reasoning Decomposition
- 将推理拆分为视觉与语言两部分,强制生成自包含的视觉描述。
- 在多个任务上显著减少视觉幻觉,提升视觉推理准确率。
- 无需外部视觉监督,训练效率更高,适合资源受限场景。
视觉语言模型(VLMs)常出现视觉幻觉:生成与视觉输入不一致的内容,或依赖语言先验跳过视觉信息。现有后训练方法多依赖可验证的答案匹配,仅监督最终输出,导致中间视觉推理缺乏显式引导,模型倾向于优先使用语言推理。本文提出Vision SR1,一种三阶段自奖励强化学习方法,通过分解推理为视觉与语言两个组件,在不依赖外部视觉监督的前提下提升视觉推理能力。模型首先被提示生成独立于图像的自包含视觉描述,再通过多奖励损失联合优化两者。为验证自包含性,同一模型被重新提示仅用生成的视觉描述进行语言推理以计算视觉奖励。最终奖励采用解耦的奖励-优势框架,分别计算视觉与语言推理的优势。实验表明,Vision SR1在多样任务中有效提升视觉推理、缓解视觉幻觉,并减少对语言捷径的依赖,且无需额外GPU,比依赖外部视觉奖励模型的方法更高效。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) often suffer from visual hallucinations: generating things that are not consistent with visual inputs and language shortcuts, where they skip the visual part and just rely on text priors. These issues arise because most post training methods for VLMs rely on simple verifiable answer matching and supervise only final outputs, leaving intermediate visual reasoning without explicit guidance. As a result, VLMs receive sparse visual signals and often learn to prioritize language based reasoning over visual perception. We introduce Vision SR1, a three stage self rewarding reinforcement learning method that improves visual reasoning without relying on external visual supervision. Vision SR1 decomposes VLM reasoning into two components: visual reasoning and language reasoning, where the model is first prompted to produce self-contained visual descriptions sufficient to answer the question without referring back to the input image, before jointly optimizing both visual and language reasoning through our multi reward loss objective. To validate this self containment, the same VLM model is reprompted to perform language reasoning using only the generated visual reasoning as input to compute visual reward. The final reward is computed through a decoupled reward-advantage framework, where visual reward and language reasoning reward each have their advantages calculated separately. Our experiments show that Vision SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts across diverse vision language tasks, while being more efficient than methods that rely on external visual reward models, which require additional GPUs to host. In contrast, Vision SR1 introduces no extra GPU overhead beyond that of standard training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。