arXiv:2602.11424cs.CLcs.AI2026-02被引 2

让梯度按可信度调整,提升微调的稳定与效果

Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives

  • 用广义对数族统一微调目标,引入可信度控制门控机制
  • 在多个数据集上显著提升模型性能,尤其在噪声标签下更稳健
  • 无需调参,适合追求稳定微调效果的研究者和工程师

标准监督微调(SFT)使用负对数似然(NLL)进行均匀的词元级加权,这种刚性导致双重缺陷:(i) 过度强调低概率目标会放大噪声监督的梯度,破坏鲁棒先验;(ii) 模型已自信时,均匀加权难以实现有效锐化。现有方法无法解决可塑性-稳定性困境,常同时抑制必要学习信号与有害信号。本文将词元级SFT目标统一于广义变形对数族中,揭示通用的‘门控×误差’梯度结构,其中门控决定模型对其当前预测的信任程度。通过凯利变换,将模型持续演化的不确定性映射到连续关注轨迹,实现新概念与已有知识场景间的无缝插值。提出无参数的动态熵微调(DEFT),以Rényi-2熵作为分布集中度的代理,调节信任门控。大量实验与分析表明,DEFT在探索与利用间取得更好平衡,整体性能显著提升。

原文摘要 · Abstract (English)

Standard negative log-likelihood (NLL) for Supervised Fine-Tuning (SFT) applies uniform token-level weighting. This rigidity creates a two-fold failure mode: (i) overemphasizing low-probability targets can amplify gradients on noisy supervision and disrupt robust priors, and (ii) uniform weighting provides weak sharpening when the model is already confident. Existing methods fail to resolve the resulting plasticity--stability dilemma, often suppressing necessary learning signals alongside harmful ones. To address this issue, we unify token-level SFT objectives within a generalized deformed-log family and expose a universal gate $\times$ error gradient structure, where the gate controls how much the model trusts its current prediction. By employing the Cayley transform, we map the model's continuously evolving uncertainty onto a continuous focus trajectory, which enables seamless interpolation between scenarios involving uncertain novel concepts and those involving well-established knowledge. We then introduce Dynamic Entropy Fine-Tuning (DEFT), a parameter-free objective that modulates the trust gate using distribution concentration (Rényi-2 entropy) as a practical proxy for the model's predictive state. Extensive experiments and analyses demonstrate that DEFT achieves a better balance between exploration and exploitation, leading to improved overall performance.

微调梯度控制熵优化自适应学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。