让大模型用更高效的离线强化学习，提升生成质量
ShiQ: Bringing back Bellman to LLMs
- 基于贝尔曼方程设计新损失函数，适配大模型特性
- 在UltraFeedback和BFCL-V3上显著优于传统方法
- 支持离线、逐标记学习，适合资源有限的场景
使用强化学习微调预训练大语言模型通常采用直接策略优化。相比之下，尽管Q-learning在非语言模型任务中表现优异，但在大模型领域却较少被关注。其优势在于样本效率和离线学习能力，对计算成本高昂的大模型尤为关键。然而，直接对模型输出概率进行类Q-learning更新无效，因大模型具有特殊性。本文从贝尔曼方程出发，推导出理论严谨的损失函数，将Q-learning适配至大模型。通过结合强化学习文献中的洞见并考虑大模型特点，确保输出logits成为可靠的Q值估计。据此提出ShiQ（Shifted-Q）算法，支持离线、逐标记的策略学习且实现简单。在合成数据及真实基准（如UltraFeedback、BFCL-V3）上的实验表明，该方法在单轮与多轮场景中均有效提升性能。
原文摘要 · Abstract (English)
The fine-tuning of pre-trained large language models (LLMs) using reinforcement learning (RL) is generally formulated as direct policy optimization. This approach was naturally favored as it efficiently improves a pretrained LLM, seen as an initial policy. Another RL paradigm, Q-learning methods, has received far less attention in the LLM community while demonstrating major success in various non-LLM RL tasks. In particular, Q-learning effectiveness comes from its sample efficiency and ability to learn offline, which is particularly valuable given the high computational cost of sampling with LLMs. However, naively applying a Q-learning-style update to the model's logits is ineffective due to the specificity of LLMs. Our core contribution is to derive theoretically grounded loss functions from Bellman equations to adapt Q-learning methods to LLMs. To do so, we carefully adapt insights from the RL literature to account for LLM-specific characteristics, ensuring that the logits become reliable Q-value estimates. We then use this loss to build a practical algorithm, ShiQ for Shifted-Q, that supports off-policy, token-wise learning while remaining simple to implement. Finally, we evaluate ShiQ on both synthetic data and real-world benchmarks, e.g., UltraFeedback and BFCL-V3, demonstrating its effectiveness in both single-turn and multi-turn LLM settings
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。