通过自适应熵调控，缓解微调时的自信冲突，防止模型遗忘通用能力。
Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
- 用词级熵值区分不确定性与知识冲突，选择性更新梯度。
- 在数学、医疗等多领域测试中，性能持平标准微调，通用能力下降减少50%以上。
- 适合需保持模型泛化能力的高阶应用，如智能代理和专业领域适配。
监督微调（SFT）是领域适配的标准范式，但常导致灾难性遗忘。相比之下，基于策略的强化学习（RL）能有效保留通用能力。我们研究发现根本原因在于分布差异：RL与模型内部信念一致，而SFT强制模型匹配外部监督。这种不匹配常表现为“自信冲突”——低概率但低熵的词元，模型高度自信却被迫学习矛盾真值，引发破坏性梯度更新。为此，我们提出熵自适应微调（EAFT）。不同于仅依赖预测概率的方法，EAFT以词元级熵为门控机制，区分认知不确定性与知识冲突。这使模型可从不确定样本学习，同时抑制冲突数据的梯度传播。在4B至32B参数的Qwen与GLM系列模型上，涵盖数学、医疗和智能体领域的大量实验验证了假设。EAFT在下游任务性能上与标准SFT相当，同时显著缓解通用能力退化，效果提升超50%。
原文摘要 · Abstract (English)
Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts" tokens characterized by low probability but low entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes token-level entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。