arXiv:2606.16429cs.LGcs.CL2026-06

提出新初始化方法,让压缩版注意力模型更快更准地继承大模型能力。

Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

论文配图:Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
图 1 · 摘自论文原文
  • 基于泰勒展开分析教师模型统计,自动设定学生模型参数
  • 零样本性能提升最高达88倍,训练成本降低4.9至9.2倍
  • 适合需要高效部署长序列模型的研究者和工程师

混合线性注意力模型为实现更快的长序列推理提供了可行路径:它们在保持Transformer模型质量的同时,降低了全softmax注意力的二次复杂度和键值缓存负担。一种实用方法是将预训练的Transformer转换为新架构,而非从头预训练。然而这种转换仍不稳定:简单复制教师模型的注意力投影到门控增量网络(GDN)学生模型中,无法确定新的循环衰减、写入和输出门控动态。结果,转换后的模型常始于不良动力学状态,需耗费大量微调令牌修复初始状态,而非学习教师行为。我们提出Taylor-Calibrate,一种轻量级的混合GDN学生模型初始化方法。该方法利用泰勒引导的教师注意力统计信息,设定值投影、记忆时标、写入门控与输出门控,并通过短时逐层对齐步骤匹配每层教师输出。在四种教师设置和三种保留层策略下,Taylor-Calibrate显著提升零样本学生模型性能,代表性消融实验中性能提升达88倍,且达到相同恢复目标所需训练令牌数减少4.9至9.2倍。

原文摘要 · Abstract (English)

Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models. A practical way to obtain such models is to convert a pretrained Transformer instead of pretraining a new architecture from scratch, but this conversion is still brittle. Simply copying the teacher attention projections into a Gated DeltaNet (GDN) student does not specify the new recurrent decay, write, and output-gating dynamics. As a result, the converted model often starts in a poor dynamical regime and must spend many distillation tokens repairing initialization rather than learning the remaining teacher behavior. We propose Taylor-Calibrate, a lightweight initialization method for hybrid GDN students. The method uses Taylor-guided teacher attention statistics to set the value projection, memory timescale, write gates, and output gate, then applies a short per-layer alignment step to match each converted layer to the teacher output. Across four teacher settings and three retained-layer policies, Taylor-Calibrate gives substantially stronger zero-shot students, with up to an 88x improvement in a representative ablation, and reaches matched recovery targets with 4.9x--9.2x fewer training tokens than naive conversion.

注意力机制模型压缩初始化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。