开源大规模LLM强化学习系统，实现AIME 2024 50分成绩
DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- 提出DAPO算法，解耦剪裁与动态采样策略提升训练效率
- 在Qwen2.5-32B上达成AIME 2024 50分，可复现性强
- 全开源训练代码与数据集，助力社区研究
推理扩展赋予大模型前所未有的推理能力，强化学习是激发复杂推理的核心技术。然而，当前先进推理型大模型的关键技术细节（如OpenAI o1博客和DeepSeek R1技术报告）仍被隐藏，导致社区难以复现其强化学习训练结果。本文提出解耦剪裁与动态采样策略优化（DAPO）算法，并完整开源一个达到前沿水平的大规模强化学习系统，使用Qwen2.5-32B基础模型在AIME 2024上获得50分。不同于以往研究隐匿训练细节的做法，我们公开了算法的四项关键技术，使大规模大模型强化学习成为可能。此外，我们基于verl框架开源了训练代码，并提供精心构建与处理的数据集。该开源系统显著提升了可复现性，为未来大规模大模型强化学习研究提供支持。
原文摘要 · Abstract (English)
Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the $\textbf{D}$ecoupled Clip and $\textbf{D}$ynamic s$\textbf{A}$mpling $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{DAPO}$) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。