arXiv:2602.21669cs.CL2026-02Conference of the …被引 4

通过双空间加权与时间扭曲对齐,提升跨分词器知识蒸馏效果。

DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation

  • 在词元和序列双空间中加权,聚焦学生不确定、教师自信的关键词元。
  • 采用软动态时间规整对齐嵌入层和隐状态层,精准匹配语义与上下文。
  • 在多个NLP任务上超越现有方法,适合大模型压缩场景使用。

知识蒸馏(KD)已成为压缩大语言模型(LLMs)的关键技术。尽管现有跨分词器KD方法已取得显著进展,其效果仍受限于序列与词汇层面的对齐不足。为此,本文提出双空间加权与时间扭曲对齐(DWA-KD),一种新型跨分词器蒸馏框架。在词元层面,DWA-KD将教师表示映射至学生空间并反向映射,通过Kullback-Leibler散度实现双空间蒸馏,并引入基于熵的双空间权重,增强学生不确定而教师自信的词元的学习强度,避免均匀处理所有位置。在序列层面,DWA-KD在嵌入层与最终隐藏状态层均应用软动态时间规整(Soft-DTW),融合词汇与语义信息,实现教师与学生序列间的鲁棒对齐。大量实验表明,DWA-KD在多个NLP基准上优于当前最优蒸馏基线,消融实验证实了熵加权与两层Soft-DTW对齐的互补贡献。

原文摘要 · Abstract (English)

Knowledge Distillation (KD) has emerged as a crucial technique for compressing Large Language Models (LLMs). Although existing cross-tokenizer KD methods have made notable progress, their effectiveness remains constrained by suboptimal alignment across sequence and vocabulary levels. To address these limitations, we introduce Dual-Space Weighting and Time-Warped Alignment (DWA-KD), a novel cross-tokenizer distillation framework that enhances token-wise distillation through dual-space entropy-based weighting and achieves precise sequence-level alignment by leveraging both lexical and semantic information. At the token level, DWA-KD maps teacher representations into the student space and vice versa, performing dual-space KD via Kullback-Leibler divergence (KL). The process is modulated by dual-space weights that up-weight tokens where the student is uncertain and the teacher is confident, thereby focusing learning on informative tokens rather than treating all positions equally. At the sequence level, DWA-KD applies Soft Dynamic Time Warping (Soft-DTW) to both the embedding and final hidden-state layers, enabling robust alignment of lexical and contextual semantics between teacher and student sequences. Extensive experiments across diverse NLP benchmarks demonstrate that DWA-KD outperforms state-of-the-art KD baselines, while ablation studies confirm the complementary contributions of entropy-based token weighting and embedding and final hidden state layer Soft-DTW alignment.

知识蒸馏大模型压缩跨分词器序列对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。