arXiv:2606.11189cs.LGcs.AI2026-06

通过设计目标分布,让微调更聪明地学习人类示范。

A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design

论文配图:A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design
图 1 · 摘自论文原文
  • 把微调看作目标分布的设计问题,而非简单复制每个词。
  • 新方法在十个推理任务中均优于现有方法。
  • 适合想改进微调效果的研究者和工程师。

监督微调(SFT)通常最大化示范轨迹中每个标记的似然,但观察到的标记可能不唯一、含噪声或与模型先验不一致。严格匹配单热目标可能导致次优结果,尤其当预训练模型已包含丰富知识先验时。本文将SFT重新理解为目标分布设计:不只关注损失函数,而是分析损失驱动模型匹配的逐标记目标。提出Q-target框架,将SFT监督分解为两个显式选择:(1)对观察标记的信任程度;(2)剩余概率质量在其他选项间的分配方式。这一视角统一了多种现有SFT变体作为隐式的目标分布选择。基于此,我们提出Target-SFT,直接从期望目标分布构建训练目标。该方法在十组推理数据集-模型设置中表现一致更优,验证了基于目标的方法的有效性。整体上,该公式揭示了SFT训练更根本的设计原则,并拓展了目标搜索空间。

原文摘要 · Abstract (English)

Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior. In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that the loss drives the model to match. We introduce the Q-target framework, which decomposes SFT supervision into two explicit choices: (1) how strongly to rely on the observed token, and (2) how to allocate the remaining probability mass over alternatives. This perspective unifies many existing SFT variants as implicit choices of the target distribution Q. Building on this view, we propose Target-SFT which constructs the training objective directly from the desired target distribution. This method consistently outperforms across the ten reasoning dataset-model settings evaluated, showing the effectiveness of this target-based approach. Overall, our formulation reveals a more fundamental design principle for SFT training and opens a broader search space for SFT objectives.

监督微调目标分布模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。