SRPO让大模型在数学与编程上更强更省，只需十分之一训练量。
SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
- 分两阶段跨领域训练，兼顾数学与编程能力提升
- 用历史重采样技术解决无效样本问题，性能超DeepSeek-R1-Zero-32B
- 仅需1/10训练步数，适合高效强化学习研究者参考
近期推理模型（如OpenAI的o1和DeepSeek的R1）表明强化学习（RL）能显著提升大语言模型（LLM）的推理能力。然而，跨领域复现这些成果仍因方法不透明而困难。本文提出两阶段历史重采样策略优化（SRPO），在AIME24和LiveCodeBench基准上超越DeepSeek-R1-Zero-32B表现。使用与DeepSeek相同的基座模型（Qwen2.5-32B），仅需其约1/10训练步数即达更优效果，体现更高效率。基于组相对策略优化（GRPO），引入两项创新：（1）双阶段跨域训练范式，平衡数学推理与编程能力发展；（2）历史重采样（HR）机制，有效处理无效样本。全面实验验证方法有效性，为扩展LLM推理能力提供新思路。
原文摘要 · Abstract (English)
Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。