提出一种融合演示与探索的微调方法，提升大模型推理能力。
Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
- 用前缀采样实现监督与强化微调的协同学习
- 在数学推理任务中性能超越单一微调及混合策略
- 对演示数据质量与数量变化具有强鲁棒性
现有大模型后训练技术主要分为监督微调（SFT）和强化微调（RFT）。SFT擅长模仿演示数据，但易导致行为克隆式泛化偏差；RFT可显著提升性能，却易学习意外行为，且对初始策略敏感。本文提出统一视角，引入Prefix-RFT，一种结合演示与探索的混合方法。以数学推理问题为测试基准，实验证明该方法简单有效：不仅优于独立的SFT与RFT，还超越并行混合策略的RFT方法。分析表明SFT与RFT具有互补性，Prefix-RFT能有效协同二者。消融实验进一步验证其对演示数据质量和数量变化的鲁棒性。
原文摘要 · Abstract (English)
Existing LLMs-post-training techniques are broadly categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Each paradigm presents a distinct trade-off: (1) SFT excels at mimicking demonstration data, but can lead to problematic generalization as a form of behavior cloning. (2) Conversely, RFT can significantly enhance a model's performance but is prone to learning unexpected behaviors, and its performance is sensitive to the initial policy. In this paper, we propose a unified view of these methods and introduce Prefix-RFT, a hybrid approach that synergizes learning from both demonstration and exploration. Using mathematical reasoning problems as a test bed, we empirically demonstrate that Prefix-RFT is simple yet effective. Not only does it surpass the performance of standalone SFT and RFT, but it also outperforms parallel mixed-policy RFT methods. Our analysis highlights the complementary nature of SFT and RFT, validating that Prefix-RFT effectively harmonizes them. Further ablation studies confirm the method's robustness to variations in the quality and quantity of demonstration data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。