arXiv:2606.21851cs.CL2026-06ACL

通过分层对齐与自适应优化,提升嵌入蒸馏的效率和性能。

TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation

论文配图:TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation
图 1 · 摘自论文原文
  • 仅在学生模型上层蒸馏教师句向量,降低计算开销。
  • 用几何关系约束实现层间知识传递,缩小语义差距。
  • 引入抗噪声优化机制,提升模型泛化能力,适合部署场景。

知识蒸馏已成为压缩大型预训练语言模型的关键技术。然而,现有方法强制学生严格模仿教师的句向量或内部特征,常因固有的容量差距导致计算成本过高且性能不佳。为此,我们提出TALAS(教师锚定分层对齐与自适应锐度感知最小化),一个融合分层对齐与鲁棒优化的统一框架。首先,引入教师锚定机制,仅将教师最终句向量蒸馏至学生模型上层,减少开销并尊重容量限制。其次,通过层对齐自蒸馏,在嵌入空间中利用内部几何关系自上而下传递知识,弥合底层语义差距。最后,为防止学生记忆教师的点级噪声,将自适应锐度感知最小化(ASAM)引入训练目标,引导模型趋向平坦极小值以增强泛化性。在标准句向量基准上的实验表明,TALAS持续优于强基线方法,同时在计算成本和内存占用方面表现出更优的训练效率。

原文摘要 · Abstract (English)

Knowledge Distillation (KD) has established itself as a pivotal technique for compressing large pre-trained language models. However, existing methods that force a student to strictly mimic the teacher's sentence embeddings or internal features often incur prohibitive computational costs and yield suboptimal performance due to the inherent capacity gap. To address these challenges, we propose TALAS (Teacher-Anchored Layer Alignment with Sharpness-aware minimization), a unified framework that synergizes hierarchical (multi-layer) alignment with robust optimization. First, we introduce a Teacher-Anchored mechanism that selectively distills final sentence embeddings only into the student's upper layers, thereby reducing overhead while respecting capacity constraints. Second, we bridge the semantic gap in lower layers via Layer-Aligned Self-Distillation, which propagates knowledge top-down using internal geometric relational constraints in the embedding space. Finally, to prevent the student from memorizing point-wise teacher noise, we integrate Adaptive Sharpness-Aware Minimization (ASAM) into the training objective, guiding the model towards flat minima for enhanced generalization. Empirical results on standard sentence embedding benchmarks demonstrate that TALAS consistently outperforms strong distillation baselines while achieving superior training efficiency in terms of computational cost and memory footprint.

知识蒸馏嵌入对齐模型压缩优化方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。