提出新理论与方法，让SFT实现类似强化学习的泛化能力
Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training
- 基于分布判别理论，量化数据与模型分布的对齐程度
- 在多个基准上超越DPO、SimPO等离线强化学习方法
- 适合无法使用RL但追求强泛化的大型语言模型训练场景
监督微调（SFT）计算高效，但泛化能力常逊于强化学习（RL），主要因后者使用了在线策略数据。本文提出框架实现在线策略SFT，首次提出分布判别理论（DDT），用于解释并量化数据分布与模型诱导分布之间的对齐程度。基于DDT，提出两种互补技术：(i) 在分布内微调（IDFT），一种损失层面的方法以提升SFT泛化能力；(ii) 提示解码（Hinted Decoding），一种数据层面的技术，可将训练语料重新对齐至模型分布。大量实验表明，该框架在泛化性能上超越主流离线强化学习算法（如DPO、SimPO），同时保持SFT的效率。该框架为无法使用强化学习的领域提供了实用替代方案。代码已开源：https://github.com/zhangmiaosen2000/Towards-On-Policy-SFT
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is computationally efficient but often yields inferior generalization compared to reinforcement learning (RL). This gap is primarily driven by RL's use of on-policy data. We propose a framework to bridge this chasm by enabling On-Policy SFT. We first present \textbf{\textit{Distribution Discriminant Theory (DDT)}}, which explains and quantifies the alignment between data and the model-induced distribution. Leveraging DDT, we introduce two complementary techniques: (i) \textbf{\textit{In-Distribution Finetuning (IDFT)}}, a loss-level method to enhance generalization ability of SFT, and (ii) \textbf{\textit{Hinted Decoding}}, a data-level technique that can re-align the training corpus to the model's distribution. Extensive experiments demonstrate that our framework achieves generalization performance surpassing prominent offline RL algorithms, including DPO and SimPO, while maintaining the efficiency of an SFT pipeline. The proposed framework thus offers a practical alternative in domains where RL is infeasible. We open-source the code here: https://github.com/zhangmiaosen2000/Towards-On-Policy-SFT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。