arXiv:2605.01374cs.CL2026-05ACL被引 1

通过分层对齐教师与学生模型的语义演化轨迹,提升大模型压缩效果。

MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation

论文配图:MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation
图 1 · 摘自论文原文
  • 按层次动态调整对齐粒度:底层对词,高层对短语
  • 在多个基准上超越现有方法,性能提升显著
  • 适合需要高效压缩大模型的部署场景

知识蒸馏是压缩大语言模型的关键技术,但现有方法通常在固定层或词级输出上对齐表示,忽略了表示随深度演化的过程。这导致学生模型难以捕捉教师模型内部的结构关系,限制了知识迁移效果。为此,我们提出多粒度轨迹对齐(MTA)框架,沿层间变换轨迹对齐教师与学生模型的表示。MTA采用分层自适应策略:低层在词级对齐以保留词汇信息,高层在短语级跨度(如名词、动词短语)上对齐以捕获组合语义。通过动态结构对齐损失,匹配每层内语义单元间的相对几何关系。该设计基于实证发现——Transformer表示随深度日益抽象,并符合语言学观点:高层语义由低层词汇单位组合生成。此外,引入隐藏表示对齐损失,直接对齐选定层的教师-学生表示。实验表明,MTA在标准基准上持续优于当前最优基线,消融实验验证了各组件的有效性。

原文摘要 · Abstract (English)

Knowledge distillation is a key technique for compressing large language models (LLMs), but most existing methods align representations at fixed layers or token-level outputs, ignoring how representations evolve across depth. As a result, the student is only weakly guided to capture the teacher's internal relational structure during distillation, which limits knowledge transfer. To address this limitation, we propose Multi-Granular Trajectory Alignment (MTA), a framework that aligns teacher and student representations along their layer-wise transformation trajectory. MTA adopts a layer-adaptive strategy: lower layers are aligned at the word level to preserve lexical information, while higher layers operate on phrase-level spans (e.g., noun and verb phrases) to capture compositional semantics. We instantiate this idea through a Dynamic Structural Alignment loss that matches the relative geometry among semantic units within each layer. This design is motivated by empirical findings that Transformer representations become increasingly abstract with depth, and is also consistent with linguistic views in which higher-level meaning emerges through the composition of lower-level lexical units. We further incorporate a Hidden Representation Alignment loss to directly align selected teacher-student layers. Experiments show that MTA consistently outperforms state-of-the-art baselines on standard benchmarks, with ablations confirming the contribution of each component.

大模型压缩知识蒸馏多粒度对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。