训练状态分布比具体指令更重要,影响模型性能与记忆保留。
Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation
- 从状态分布角度重新理解微调、强化学习和蒸馏方法
- 轻量级强化学习提升解题能力且不丢失原有知识
- 用退化教师模型蒸馏反而表现更好,说明状态来源关键
大型语言模型的后训练方法如监督微调(SFT)、强化学习(RL)和在线策略蒸馏(OPD)通常从损失函数角度分析:最大似然、策略梯度、前向KL、反向KL等。本文提出互补视角:监督所基于的状态分布。对于自回归策略,状态指提示词加已生成前缀。SFT在固定数据集状态上训练,而RL与OPD则在当前学习者生成的状态上训练。我们将后训练建模为状态分布塑造,并在Qwen3-0.6B-Base上进行小规模控制实验,使用GSM8K作为训练任务,TruthfulQA和MMLU作为保留评估指标。结果发现三个现象:第一,轻微SFT可提升GSM8K表现且遗忘极少;重度SFT则导致显著保留损失。第二,从退化SFT教师模型出发的OPD,在GSM8K、TruthfulQA和MMLU上均超越教师模型,尽管仅以教师为唯一监督源。第三,轻量级在线策略强化学习在提升GSM8K表现的同时保持了知识保留。这些结果支持一种以状态为中心的后训练观:训练状态的来源与局部性可能与监督信号形式同样重要。
原文摘要 · Abstract (English)
Large language model post-training methods such as supervised fine-tuning (SFT), reinforcement learning (RL), and distillation are often analyzed through their loss functions: maximum likelihood, policy gradients, forward KL, reverse KL, or related objective-level variants. We study a complementary factor: the state distribution on which supervision is applied. For an autoregressive policy, a state is a prompt plus generated prefix. SFT trains on fixed dataset states, while RL and on-policy distillation (OPD) train on states induced by the current learner. We formalize post-training as state-distribution shaping and run a controlled smallscale study using Qwen3-0.6B-Base on GSM8K, with TruthfulQA and MMLU as retention evaluations. Our results show three phenomena. First, a mild SFT run improves GSM8K with little forgetting, while a stress SFT run causes substantial retention loss. Second, OPD from a degraded SFT teacher surpasses that teacher on GSM8K, TruthfulQA, and MMLU, despite using the teacher as its only supervision source. Third, a lightweight on-policy RL run improves GSM8K while preserving retention. These results support a state-centric view of post-training: the source and locality of training states can be as important as the form of the supervision signal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。