对比三种强化学习算法，提升大模型复杂推理能力。
Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement
- 在计时游戏上微调后，用通用推理基准评估三种算法
- 增大分组规模可稳定训练并提高准确率，KL惩罚非单调影响
- DAPO中动态采样反而降低效果，关闭它表现最佳
本研究系统比较了三种强化学习算法（PPO、GRPO 和 DAPO）在提升大语言模型复杂推理能力方面的表现。主要贡献在于控制性迁移学习评估：模型先在专用的计时游戏任务上微调，再在一系列通用推理基准上测试。所有任务中，经强化学习训练的模型均优于对应基础模型，但提升幅度因基准而异。参数分析表明，增加 GRPO 与 DAPO 的分组规模可带来更稳定的训练过程和更高的准确率；而 KL 惩罚系数的影响并非单调。此外，我们发现 DAPO 中的动态采样（DS）组件并未提升性能，实际上在关闭 DS 时取得最佳整体结果。
原文摘要 · Abstract (English)
This study presents a systematic comparison of three Reinforcement Learning (RL) algorithms (PPO, GRPO, and DAPO) for improving complex reasoning in large language models (LLMs). Our main contribution is a controlled transfer-learning evaluation: models are first fine-tuned on the specialized Countdown Game and then assessed on a suite of general-purpose reasoning benchmarks. Across all tasks, RL-trained models outperform their corresponding base models, although the degree of improvement differs by benchmark. Our parametric analysis offers practical guidance for RL-based LLM training. Increasing the group size in GRPO and DAPO leads to more stable training dynamics and higher accuracy, while the impact of the KL-penalty coefficient is non-monotonic. Additionally, we find that the Dynamic Sampling (DS) component in DAPO does not improve performance; in fact, the best overall results are achieved with DAPO when DS is disabled.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。