用轨迹贝尔曼残差优化，让大模型推理更高效。
Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
- 直接用模型输出概率作为价值函数，不依赖评判器
- 单次采样即可训练，性能超越PPO等策略方法
- 适合追求简单高效推理增强的研究者
当前大语言模型（LLM）推理中的强化学习主要采用基于策略的方法，而基于价值的方法尚未被充分探索。本文重新审视经典的贝尔曼残差最小化思想，提出轨迹贝尔曼残差最小化（TBRM），一种自然适配于LLM的简单有效离策略算法。该方法仅需一次提示采样，利用模型自身的logits作为$Q$-值，优化单条轨迹上的贝尔曼目标，无需评判器、重要性采样比率或截断。我们通过改进的轨迹测度变换分析，证明其可从任意离策略数据收敛至近似最优的KL正则化策略。在标准数学推理基准上的实验表明，TBRM持续优于如PPO和GRPO等策略基基线，计算与内存开销相当或更低。结果表明，基于价值的强化学习或可成为提升大模型推理能力的一种原理性强且高效的替代方案。
原文摘要 · Abstract (English)
Policy-based methods currently dominate reinforcement learning (RL) pipelines for large language model (LLM) reasoning, leaving value-based approaches largely unexplored. We revisit the classical paradigm of Bellman Residual Minimization and introduce Trajectory Bellman Residual Minimization (TBRM), an algorithm that naturally adapts this idea to LLMs, yielding a simple yet effective off-policy algorithm that optimizes a single trajectory-level Bellman objective using the model's own logits as $Q$-values. TBRM removes the need for critics, importance-sampling ratios, or clipping, and operates with only one rollout per prompt. We prove convergence to the near-optimal KL-regularized policy from arbitrary off-policy data via an improved change-of-trajectory-measure analysis. Experiments on standard mathematical-reasoning benchmarks show that TBRM consistently outperforms policy-based baselines, like PPO and GRPO, with comparable or lower computational and memory overhead. Our results indicate that value-based RL might be a principled and efficient alternative for enhancing reasoning capabilities in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。