用变分推理统一语言模型的思维过程与强化学习，提升推理稳定性。
Variational Reasoning for Language Models
- 将思考轨迹视为隐变量，通过变分推断优化推理路径。
- 提出前向KL公式，训练更稳定，且自然产生对简单问题的偏好偏差。
- 实验证明在Qwen系列模型上有效，适用于多种推理任务。
我们提出一种变分推理框架，将语言模型的思考轨迹视为隐变量，并通过变分推断进行优化。从证据下界（ELBO）出发，扩展为多轨迹目标以获得更紧的界限，并提出前向KL形式，稳定了变分后验的训练。我们进一步证明拒绝采样微调和二元奖励强化学习（包括GRPO）可被解释为局部前向KL目标，其推导中自然出现模型准确率的隐式加权，揭示了对较简单问题的未被注意的偏好偏差。我们在Qwen 2.5和Qwen 3模型家族上广泛验证了该方法，结果表明该工作提供了统一变分推断与强化学习方法的原理性视角，生成了稳定的目标，有效提升了语言模型的推理能力。代码已公开于https://github.com/sail-sg/variational-reasoning。
原文摘要 · Abstract (English)
We introduce a variational reasoning framework for language models that treats thinking traces as latent variables and optimizes them through variational inference. Starting from the evidence lower bound (ELBO), we extend it to a multi-trace objective for tighter bounds and propose a forward-KL formulation that stabilizes the training of the variational posterior. We further show that rejection sampling finetuning and binary-reward RL, including GRPO, can be interpreted as local forward-KL objectives, where an implicit weighting by model accuracy naturally arises from the derivation and reveals a previously unnoticed bias toward easier questions. We empirically validate our method on the Qwen 2.5 and Qwen 3 model families across a wide range of reasoning tasks. Overall, our work provides a principled probabilistic perspective that unifies variational inference with RL-style methods and yields stable objectives for improving the reasoning ability of language models. Our code is available at https://github.com/sail-sg/variational-reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。