用小模型提升大模型推理效率，减少冗余思考
REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoning
- 引入小型反思模型实现在线高效训练，支持并行采样与顺序修正
- 设计反思奖励机制，避免过度简化导致能力下降，推理成本降36%
- 适合追求高效率与强推理能力平衡的复杂任务应用
大型推理模型在复杂任务中表现优异，但常因过度思考导致推理成本过高。现有方法通过生成和过滤数据来学习简短推理路径，但在线使用时效率低下。而主流在线强化学习采用长度奖励鼓励简洁输出，却易丧失反思能力并损害性能。为此，我们提出REA-RL，引入小型反思模型实现在线训练的高效扩展，支持并行采样与序列修订；同时设计反思奖励，防止模型偏好短但缺乏反思的响应。实验表明，两种方法均在保持或提升性能的同时显著提高推理效率，组合后使推理成本降低36%且不损失性能。进一步分析显示，该方法在难题上维持高反思频率，在简单问题上适度降低反思频率而不失反思能力。代码已开源。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) demonstrate strong performance in complex tasks but often face the challenge of overthinking, leading to substantially high inference costs. Existing approaches synthesize shorter reasoning responses for LRMs to learn, but are inefficient for online usage due to the time-consuming data generation and filtering processes. Meanwhile, online reinforcement learning mainly adopts a length reward to encourage short reasoning responses, but it tends to lose reflection ability and harm performance. To address these issues, we propose REA-RL, which introduces a small reflection model for efficient scaling in online training, offering both parallel sampling and sequential revision. Besides, a reflection reward is designed to further prevent LRMs from favoring short yet non-reflective responses. Experiments show that both methods maintain or enhance performance while significantly improving inference efficiency. Their combination achieves a good balance between performance and efficiency, reducing inference costs by 36% without compromising performance. Further analysis demonstrates that our methods are effective by maintaining reflection frequency for hard problems while appropriately reducing it for easier ones without losing reflection ability. Code is available at https://github.com/hexuandeng/REA-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。