arXiv:2510.19266cs.LG2025-10被引 3

用注意力桥实现跨架构高效知识蒸馏,让状态空间模型快速学习Transformer能力。

Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge

  • 设计轻量注意力桥,实现教师模型到学生模型的逐标记监督
  • 在有限数据下显著提升状态空间模型性能,优于传统蒸馏方法
  • 适合想快速提升SSM性能的研究者,尤其适用于资源受限场景

状态空间模型(SSMs)作为序列建模的高效替代方案,凭借递归结构具备更优可扩展性,但其训练成本高,生态远不如Transformer成熟。且两者结构差异大,难以高效蒸馏预训练注意力模型的知识。本文提出跨架构注意力桥蒸馏(CAB),一种数据高效的蒸馏框架,可将Transformer教师模型的注意力知识有效迁移至状态空间学生模型。与仅在输出层传递知识的传统方法不同,CAB通过轻量级桥接模块和灵活的层间对齐策略,实现逐标记监督,提升效率与迁移能力。在视觉与语言任务上的大量实验表明,该方法在数据有限时仍能持续提升状态空间模型表现,优于标准及跨架构蒸馏方法。结果表明,注意力知识可高效迁移到循环模型中,助力快速利用Transformer经验构建更强的SSM社区。

原文摘要 · Abstract (English)

State-space models (SSMs) have emerged as efficient alternatives to Transformers for sequence modeling, offering superior scalability through recurrent structures. However, their training remains costly and the ecosystem around them is far less mature than that of Transformers. Moreover, the structural heterogeneity between SSMs and Transformers makes it challenging to efficiently distill knowledge from pretrained attention models. In this work, we propose Cross-architecture distillation via Attention Bridge (CAB), a novel data-efficient distillation framework that efficiently transfers attention knowledge from Transformer teachers to state-space student models. Unlike conventional knowledge distillation that transfers knowledge only at the output level, CAB enables token-level supervision via a lightweight bridge and flexible layer-wise alignment, improving both efficiency and transferability. We further introduce flexible layer-wise alignment strategies to accommodate architectural discrepancies between teacher and student. Extensive experiments across vision and language domains demonstrate that our method consistently improves the performance of state-space models, even under limited training data, outperforming both standard and cross-architecture distillation methods. Our findings suggest that attention-based knowledge can be efficiently transferred to recurrent models, enabling rapid utilization of Transformer expertise for building a stronger SSM community.

知识蒸馏状态空间模型跨架构高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。