系统梳理大模型强化学习技术,揭示对齐与推理的关键方法与瓶颈。
A Technical Survey of Reinforcement Learning Techniques for Large Language Models
- 整合PPO、RLHF、DPO等主流算法,构建技术框架
- 发现奖励劫持、计算成本高是主要挑战,验证性奖励提升推理能力
- 适合研究强化学习对齐与推理的学者参考
本综述全面介绍强化学习与语言模型的融合基础,重点分析近端策略优化(PPO)、Q-Learning、Actor-Critic等算法。深入探讨专用于大模型的RL技术,包括基于人类反馈的强化学习(RLHF)、AI反馈(RLAIF),以及直接偏好优化(DPO)和组相对策略优化(GRPO)等进阶策略。系统分析其在代码生成、工具增强推理等领域的应用。超越简单分类,从算法层面剖析策略优化中的失败模式,数学刻画结构瓶颈与稳定性权衡。提出基于奖励建模、反馈机制与优化策略的对比分类体系。评估显示,RLHF在对齐任务中仍占主导,基于结果的强化学习如可验证奖励强化学习(RLVR)显著提升步骤式推理能力。但奖励劫持、计算开销与可扩展反馈收集仍是持续挑战。还解析了近期基准表现的因果因素,区分架构扩展与特定优化目标带来的收益。展望混合强化学习、验证器引导训练与多目标对齐框架等新兴方向。本综述为推动基于强化学习的大模型发展提供路线图,兼顾能力提升、安全与可扩展性。
原文摘要 · Abstract (English)
This survey offers a comprehensive foundation on the integration of RL with language models, highlighting prominent algorithms such as Proximal Policy Optimization (PPO), Q-Learning, and Actor-Critic methods. Additionally, it provides an extensive technical overview of RL techniques specifically tailored for LLMs, including foundational methods like Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF), as well as advanced strategies such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO). We systematically analyze their applications across domains, i.e., from code generation to tool-augmented reasoning. Crucially, we move beyond descriptive categorization to provide a rigorous algorithmic analysis of failure modes, mathematically framing the structural bottlenecks and stability trade-offs inherent in policy optimization. We also present a comparative taxonomy based on reward modeling, feedback mechanisms, and optimization strategies. Our evaluation highlights key trends. RLHF remains dominant for alignment, and outcome-based RL such as Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves stepwise reasoning. However, persistent challenges such as reward hacking, computational costs, and scalable feedback collection underscore the need for continued innovation. We also explicate the causal factors behind recent benchmark performances, distinguishing between gains derived from architectural scaling versus those stemming from specific optimization objectives. We further discuss emerging directions, including hybrid RL algorithms, verifier-guided training, and multi-objective alignment frameworks. This survey serves as a roadmap for researchers advancing RL-driven LLM development, balancing capability enhancement with safety and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。