用变分推断让视觉模型推理更可靠，不依赖有偏奖励模型。
Latent Chain-of-Thought for Visual Reasoning
- 将视觉推理建模为后验推断，用稀疏奖励信号训练多样化的隐式思维链。
- 在7个基准上提升模型效果，通用性与可解释性均优于当前最佳方法。
- 适合关注大模型可解释性、推理泛化能力的研究者和开发者。
思维链（CoT）推理对提升大视觉语言模型（LVLMs）的可解释性和可靠性至关重要。然而，现有训练方法如SFT、PPO和GRPO在未见过的推理任务上泛化能力不足，且严重依赖有偏的奖励模型。为此，我们把LVLM中的推理重新建模为后验推断，并提出一种基于近似变分推断的可扩展训练算法。通过引入多样性导向的强化学习算法，设计了一种新型的逐标记稀疏奖励函数，激励生成多样且高似然的隐式思维链，克服确定性采样的局限并避免奖励欺骗。此外，我们实现了一种贝叶斯推断缩放策略，用边际似然替代昂贵的Best-of-N和束搜索，高效排序最优推理路径与答案。实验证明，该方法在7个推理基准上显著提升了当前最优LVLM的表现，在有效性、泛化性和可解释性方面均有优势。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs). However, existing training algorithms such as SFT, PPO, and GRPO may not generalize well across unseen reasoning tasks and heavily rely on a biased reward model. To address this challenge, we reformulate reasoning in LVLMs as posterior inference and propose a scalable training algorithm based on amortized variational inference. By leveraging diversity-seeking reinforcement learning algorithms, we introduce a novel sparse reward function for token-level learning signals that encourage diverse, high-likelihood latent CoT, overcoming deterministic sampling limitations and avoiding reward hacking. Additionally, we implement a Bayesian inference-scaling strategy that replaces costly Best-of-N and Beam Search with a marginal likelihood to efficiently rank optimal rationales and answers. We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on seven reasoning benchmarks, in terms of effectiveness, generalization, and interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。