提出GPPO策略,让模型更高效地学习复杂推理任务。
Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
- 用梯度保全的裁剪优化提升强化学习中的探索能力。
- 在AIME2024上达90.5%,LiveCodeBench V5上66.0%。
- 适合研究推理模型训练与强化学习优化的工程师。
我们提出Klear-Reasoner,一种具备长链推理能力的模型,在问题求解中表现出审慎思考,跨多个基准测试表现优异。尽管已有大量优秀推理模型工作,但高性能模型复现困难,常因训练细节披露不全。本文深入分析推理模型全流程,涵盖数据准备、长链思维监督微调(long CoT SFT)及强化学习(RL),并开展详尽消融实验。实验表明:少量高质量数据源比大量多样数据更有效,且难例无需准确率过滤即可取得更好效果。针对当前裁剪机制抑制探索信号和忽略次优轨迹的问题,我们提出梯度保全裁剪策略优化(GPPO),能温和回传被裁剪词元的梯度。该方法不仅增强模型探索能力,还提升对负样本的学习效率。Klear-Reasoner在数学与编程任务中表现卓越:于AIME 2024得90.5%,AIME 2025得83.2%,LiveCodeBench V5得66.0%,LiveCodeBench V6得58.1%。
原文摘要 · Abstract (English)
We present Klear-Reasoner, a model with long reasoning capabilities that demonstrates careful deliberation during problem solving, achieving outstanding performance across multiple benchmarks. Although there are already many excellent works related to inference models in the current community, there are still many problems with reproducing high-performance inference models due to incomplete disclosure of training details. This report provides an in-depth analysis of the reasoning model, covering the entire post-training workflow from data preparation and long Chain-of-Thought supervised fine-tuning (long CoT SFT) to reinforcement learning (RL), along with detailed ablation studies for each experimental component. For SFT data, our experiments show that a small number of high-quality data sources are more effective than a large number of diverse data sources, and that difficult samples can achieve better results without accuracy filtering. In addition, we investigate two key issues with current clipping mechanisms in RL: Clipping suppresses critical exploration signals and ignores suboptimal trajectories. To address these challenges, we propose Gradient-Preserving clipping Policy Optimization (GPPO) that gently backpropagates gradients from clipped tokens. GPPO not only enhances the model's exploration capacity but also improves its efficiency in learning from negative samples. Klear-Reasoner exhibits exceptional reasoning abilities in mathematics and programming, scoring 90.5% on AIME 2024, 83.2% on AIME 2025, 66.0% on LiveCodeBench V5 and 58.1% on LiveCodeBench V6.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。