不依赖学习值函数的策略优化方法在长序列任务中表现不佳，仅在短任务中可替代传统方法。
Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
- 用轨迹组间相对比较代替学习值函数估计优势
- 长周期任务中无学值函数的方法性能显著低于PPO
- 小批次分组更有效，适合短期任务场景
组相对策略优化（GRPO）通过取消学习值函数，改用轨迹组间的相对比较来估计优势，成为一种可扩展的PPO替代方案。我们在经典单任务强化学习环境中首次系统研究了GRPO，涵盖离散与连续控制任务。通过控制消融实验，发现：（1）长周期任务中学习值函数仍必不可少，所有无学值函数基线在除CartPole外的任务中均劣于PPO；（2）GRPO在高折扣因子（gamma = 0.99）下表现更好，但HalfCheetah因缺乏早期终止，中等折扣（gamma = 0.9）更优；（3）较小的组大小优于大组，表明批处理分组策略存在混杂无关轨迹的问题。结果揭示了无学值函数方法在经典控制中的局限性及其适用边界。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has emerged as a scalable alternative to Proximal Policy Optimization (PPO) by eliminating the learned critic and instead estimating advantages through group-relative comparisons of trajectories. This simplification raises fundamental questions about the necessity of learned baselines in policy-gradient methods. We present the first systematic study of GRPO in classical single-task reinforcement learning environments, spanning discrete and continuous control tasks. Through controlled ablations isolating baselines, discounting, and group sampling, we reveal three key findings: (1) learned critics remain essential for long-horizon tasks: all critic-free baselines underperform PPO except in short-horizon environments like CartPole where episodic returns can be effective; (2) GRPO benefits from high discount factors (gamma = 0.99) except in HalfCheetah, where lack of early termination favors moderate discounting (gamma = 0.9); (3) smaller group sizes outperform larger ones, suggesting limitations in batch-based grouping strategies that mix unrelated episodes. These results reveal both the limitations of critic-free methods in classical control and the specific conditions where they remain viable alternatives to learned value functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。