arXiv:2605.14967cs.LGstat.ML2026-05

通过信息感知权重,让大模型学得更高效、忘得更少。

InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting

论文配图:InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
图 1 · 摘自论文原文
  • 基于令牌信息量和置信度动态加权损失,聚焦高价值学习信号
  • 在数学、代码等任务上泛化能力优于标准微调,且保留旧能力
  • 仅需一行代码修改,适用于多种模型,适合追求稳定增量学习的场景

监督微调(SFT)是利用离线专家演示教大模型新行为的标准方法。然而,标准SFT对所有样本一视同仁——包括那些在基础模型下概率很低的样本——这会过度驱动训练更新,导致对特定样本过拟合,而非真正学习目标行为。此外,适应这些低概率样本会引发显著策略变化,损害已有能力。现有方法通过过滤、重生成或降权低概率数据缓解此问题,但常抑制了基础模型尚未掌握的新行为。我们提出InfoSFT,一种基于信息感知的SFT目标加权方案,将学习信号集中于信息量最大、置信度适中的令牌——既非过于熟悉,也非过于罕见以避免不稳。仅需对标准令牌级损失做一行修改,InfoSFT在数学、代码和思维链任务上,跨多种模型家族均显著提升泛化性能,优于原始SFT和似然加权基线,同时更好保留原有能力。

原文摘要 · Abstract (English)

Supervised fine-tuning (SFT) provides the standard approach for teaching LLMs new behaviors from offline expert demonstrations. However, standard SFT uniformly fits all samples -- including those with low likelihood under the base model -- which can disproportionately drive training updates toward overfitting specific samples rather than learning the target behavior. Moreover, adapting to these unlikely samples induces substantial policy shifts that degrade prior capabilities. Existing methods mitigate this by filtering, regenerating, or down-weighting low-likelihood data. In doing so, they often suppress precisely the novel behaviors the base model has yet to learn. We propose InfoSFT, a principled weighting scheme for the SFT objective that concentrates learning signals on maximally informative, medium-confidence tokens -- those neither overly familiar to the base model nor too unlikely to cause instability. Requiring only a one-line modification to the standard token-wise loss, InfoSFT demonstrably improves generalization over vanilla SFT and likelihood-weighted baselines across math, code, and chain-of-thought tasks with diverse model families, while better preserving pre-existing capabilities.

大模型微调信息加权知识保持强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。