融合监督与强化学习，单阶段提升大模型推理能力
SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
- 用熵感知权重统一监督与强化微调，避免两阶段流程
- 在5个数学推理数据集上达59.1%准确率，比零强化方法高9.0%
- 适合需要强推理能力的AI系统研发人员参考
大型语言模型在推理任务中表现卓越，但监督微调（SFT）与强化学习（RL）的最佳整合仍是一大挑战。通过分析词元分布、学习动态及基于熵的整合机制，我们发现SFT引起模型策略分布的粗粒度全局变化，而RL则实现细粒度的选择性优化，熵是训练有效性的关键指标。基于此，我们提出单阶段的监督强化微调（SRFT），通过熵感知加权机制统一两种微调范式。该方法直接利用示范数据和自探索回滚优化模型，而非分两阶段进行。大量实验表明，SRFT在五个数学推理基准上平均准确率达59.1%，比零强化方法提升9.0%；在三个分布外基准上提升10.9%。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspectives, we reveal key differences between these paradigms: SFT induces coarse-grained global changes to LLM policy distributions, while RL performs fine-grained selective optimizations, with entropy serving as a critical indicator of training effectiveness. Building on these observations, we propose Supervised Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach simultaneously applies SFT and RL to directly optimize the LLM using demonstrations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT achieves 59.1% average accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning benchmarks and 10.9% on three out-of-distribution benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。