揭示强化学习提升大模型规划能力的原理与局限
Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective
- 用图抽象分析强化学习在大模型规划中的机制
- 发现策略梯度法会因探索不足导致输出多样性下降
- 提出奖励设计需谨慎，避免值函数偏差
近期强化学习（RL）方法显著提升了大语言模型（LLMs）的规划能力，但其理论基础仍不清晰。本文通过可处理的图抽象框架，研究了策略梯度（PG）和Q-learning方法的优劣。理论分析表明，监督微调（SFT）可能引入基于共现的虚假解，而RL主要依赖探索实现正确规划，凸显探索对泛化能力的关键作用。然而，我们发现策略梯度存在多样性坍缩问题：训练中输出多样性下降，即使准确率已达完美仍无法恢复。相比之下，Q-learning具有两个优势：支持离策略学习，且在收敛时能保持输出多样性。此外，我们证明了奖励设计不当会导致Q值偏差。最后，在真实世界规划基准Blocksworld上的实验验证了这些现象在实践中确实存在。
原文摘要 · Abstract (English)
Recent reinforcement learning (RL) methods have substantially enhanced the planning capabilities of Large Language Models (LLMs), yet the theoretical basis for their effectiveness remains elusive. In this work, we investigate RL's benefits and limitations through a tractable graph-based abstraction, focusing on policy gradient (PG) and Q-learning methods. Our theoretical analyses reveal that supervised fine-tuning (SFT) may introduce co-occurrence-based spurious solutions, whereas RL achieves correct planning primarily through exploration, underscoring exploration's role in enabling better generalization. However, we also show that PG suffers from diversity collapse, where output diversity decreases during training and persists even after perfect accuracy is attained. By contrast, Q-learning provides two key advantages: off-policy learning and diversity preservation at convergence. We further demonstrate that careful reward design is necessary to prevent Q-value bias in Q-learning. Finally, applying our framework to the real-world planning benchmark Blocksworld, we confirm that these behaviors manifest in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。