用直接优化Q函数的方法提升大模型的多步推理能力。
Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization
- 将生成过程建模为马尔可夫决策过程，用SAC框架直接优化语言模型的Q函数。
- 在GSM8K和MATH数据集上超越已有方法，多步推理准确率显著提升。
- 适合需要长链条逻辑推理的任务，如数学解题，且无需大量在线采样。
强化学习在对齐大语言模型与人类偏好、提升其完成复杂任务的能力方面发挥关键作用。然而，现有方法要么因需多个模型和大量在线采样训练（如PPO）而消耗巨大算力，要么被建模为老虎机问题（如DPO、DRO），在涉及长思维链的多步推理任务（如数学解题）中表现不佳。为此，我们提出直接Q函数优化（DQO），将响应生成过程建模为马尔可夫决策过程（MDP），并利用软策略梯度（SAC）框架直接优化由语言模型参数化的Q函数。DQO的MDP结构相比基于老虎机的方法具有更优的流程监督能力。在GSM8K和MATH两个数学求解数据集上的实验结果表明，DQO优于以往方法，展现出作为离线强化学习对齐语言模型的潜力。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) plays a crucial role in aligning large language models (LLMs) with human preferences and improving their ability to perform complex tasks. However, current approaches either require significant computational resources due to the use of multiple models and extensive online sampling for training (e.g., PPO) or are framed as bandit problems (e.g., DPO, DRO), which often struggle with multi-step reasoning tasks, such as math problem solving and complex reasoning that involve long chains of thought. To overcome these limitations, we introduce Direct Q-function Optimization (DQO), which formulates the response generation process as a Markov Decision Process (MDP) and utilizes the soft actor-critic (SAC) framework to optimize a Q-function directly parameterized by the language model. The MDP formulation of DQO offers structural advantages over bandit-based methods, enabling more effective process supervision. Experimental results on two math problem-solving datasets, GSM8K and MATH, demonstrate that DQO outperforms previous methods, establishing it as a promising offline reinforcement learning approach for aligning language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。