小规模微调在强化学习中效果显著，但效率低，新方法可大幅提升性能。
Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning
- 提出重蒸馏技术，从强化学习策略中采样优化微调
- 用1000个样本实现超越大模型的推理性能
- 适合追求高效推理增强的研究者与工程师
R1式强化学习显著提升大语言模型的推理能力，但其机制仍不清晰。我们发现小规模监督微调（SFT）对强化学习有重要影响，但效率低下。为此，我们构建分析框架，通过测量样本效应比较SFT与RL效率。理论分析表明可提升SFT效率。基于此，我们提出重蒸馏方法，从强化学习训练好的策略中采样用于微调。该方法在三个数据集及Qwen与Llama模型上均表现惊人效率：使用更少样本和计算量即达到与强化学习相当的性能。例如，在K&K数据集上，仅用1000个SFT样本的重蒸馏Qwen-2.5-1.5B模型就超越了DeepSeek-V3-0324。实验还证明重蒸馏能高效平衡多目标强化学习。本工作解释了R1式强化学习中的多个现象，揭示其成功背后的机制。代码已开源。
原文摘要 · Abstract (English)
R1-style Reinforcement Learning (RL) significantly enhances Large Language Models' reasoning capabilities, yet the mechanism behind rule-based RL remains unclear. We found that small-scale SFT has substantial influence on RL but shows poor efficiency. To explain our observations, we propose an analytical framework and compare the efficiency of SFT and RL by measuring \textbf{sample effect}. Our hypothetical analysis shows the potential to improve SFT efficiency. Guided by our analysis, we propose \textbf{Re-distillation}, a technique that aims to boost the effectiveness of small-scale distillation by sampling from the RL-trained policy. Re-distillation shows consistent surprising efficiency on three datasets and both Qwen\&Llama models: Re-distilled models matched RL performance with far fewer samples and less computation. As a result, on K\&K dataset, our re-distilled Qwen-2.5-1.5B model surpasses DeepSeek-V3-0324 with only 1K SFT samples. We demonstrate that re-distillation can be used to efficiently balance multiple goals in RL. Our work explains several interesting phenomena in R1-style RL, shedding light on the mechanisms behind its empirical success. Code is available at: https://github.com/on1262/deep-reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。