通过探索-过滤-重放机制提升大模型推理能力，显著改善训练稳定性与效果。
EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
- 引入探索-过滤-重放三阶段框架，增强策略探索深度与效率
- 在Geometry3K上相比GRPO提升37.9%的推理准确率
- 支持样本细粒度分类与熵控，适合复杂推理任务研究者使用
近期强化学习进展显著提升了大语言模型（LLMs）的推理能力。群组相对策略优化（GRPO）作为近端策略优化（PPO）的轻量变体，虽提升了效率，但存在探索不足与训练不稳问题，限制了其在复杂推理任务中的表现。为解决此问题，我们提出EFRame——一种探索-过滤-重放框架，从三个维度增强GRPO：增加采样轮次以实现更深入、定向的探索，在线过滤低质量样本以稳定梯度并加速训练，经验回放则放大罕见但高信息量的轨迹，促进收敛稳定。该统一框架构建了兼顾探索性、效率与稳定性的训练循环。在多种推理基准上的实验表明，EFRame持续取得提升，在Geometry3K上相较GRPO实现37.9%的相对改进。此外，该方法支持细粒度样本分类与精确熵控制，展现出作为大模型深层推理推进方案的鲁棒性。代码已公开于https://github.com/597358816/EFRame。
原文摘要 · Abstract (English)
Recent advances in reinforcement learning (RL) have significantly enhanced the reasoning capabilities of large language models (LLMs). Group Relative Policy Optimization (GRPO), a lightweight variant of Proximal Policy Optimization (PPO), improves efficiency but suffers from limited exploration and training instability, limiting its effectiveness on complex reasoning tasks. To address these challenges, we introduce EFRame, an Exploration-Filter-Replay framework that augments GRPO across three dimensions: additional rollouts enable deeper and more targeted exploration, online filtering removes low-quality samples to stabilize gradients and accelerate training, and experience replay amplifies rare yet informative trajectories for stable convergence. This unified framework establishes a principled training cycle that balances exploration, efficiency, and stability. Experiments on diverse reasoning benchmarks demonstrate that EFRame achieves consistent gains, including a 37.9\% relative improvement on Geometry3K over GRPO. EFRame further supports fine-grained sample categorization and precise entropy control, highlighting it as a robust solution for advancing deeper reasoning in LLMs. Our code is available at https://github.com/597358816/EFRame.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。