arXiv:2510.11615cs.CLcs.AI2025-10被引 12

让知识蒸馏随每个词的学习状态动态调整,提升小模型性能。

LLM-Oriented Token-Adaptive Knowledge Distillation

  • 根据每个词的学习难易度动态调整蒸馏重点和温度。
  • 在多个模型和数据集上,显著优于传统静态蒸馏方法。
  • 适合需要高效压缩大模型的开发者快速部署使用。

知识蒸馏是压缩大规模语言模型的关键技术,但现有基于输出概率的方法通常采用静态策略,与学生模型的动态学习过程不匹配。这些方法对所有词一视同仁,使用固定温度,导致知识迁移效果不佳。为此,我们提出面向大模型的令牌自适应知识蒸馏(AdaKD),该框架根据每个令牌的实时学习状态动态调整蒸馏过程。AdaKD包含两个协同模块:首先,损失驱动的自适应令牌聚焦(LATF)模块通过监测学生模型的学习稳定性,动态分配计算资源,聚焦于各训练阶段最具价值的令牌;其次,提出反向难度温度缩放(IDTS),一种反直觉却有效的令牌级温度策略:对困难令牌使用低温度以精准纠错,对简单令牌使用高温度以鼓励学习教师模型完整平滑的输出分布,从而增强泛化能力。作为即插即用框架,AdaKD可一致提升多种蒸馏方法在多个模型架构和基准上的表现。

原文摘要 · Abstract (English)

Knowledge distillation (KD) is a key technique for compressing large-scale language models (LLMs), yet prevailing logit-based methods typically employ static strategies that are misaligned with the dynamic learning process of student models. These methods typically treat all tokens indiscriminately and apply a single, fixed temperature, resulting in suboptimal knowledge transfer. To address these limitations, we propose LLM-Oriented Token-Adaptive Knowledge Distillation (AdaKD), a novel framework that adapts the distillation process to the real-time learning state of each token. AdaKD consists of two synergistic modules driven by a unified token difficulty metric. First, our Loss-Driven Adaptive Token Focusing (LATF) module dynamically adjusts the distillation focus by monitoring the student's learning stability, concentrating computational resources on the most valuable tokens at each training phase. Second, we introduce Inverse Difficulty Temperature Scaling (IDTS), a counterintuitive yet effective token-level temperature strategy. It employs low temperatures for difficult tokens for targeted error correction, and high temperatures for easy tokens to encourage students to learn from the teacher's complete and smooth output distribution, thereby enhancing generalization. As a plug-and-play framework, AdaKD can consistently improve the performance of various distillation methods on multiple model architectures and benchmarks.

知识蒸馏大模型压缩自适应机制生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。