用蒸馏+贪心替换，快速构建高效混合注意力模型。
Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction
- 先蒸馏全注意力权重到线性注意力模块，再逐层替换并监控性能。
- 在多个下游任务上实现90%以上精度，推理速度提升3倍以上。
- 无需重训练或搜索，适配任意预训练模型的高效改造。
Transformer架构通过密集全注意力机制达到顶尖准确率，但其时间与内存复杂度随序列长度呈二次增长，限制了实际部署。线性注意力机制虽能实现线性或近似线性扩展，却常导致性能下降。混合模型结合全注意力与线性注意力层，有望兼顾效率与表达能力，但存在两大挑战：从头训练混合模型计算成本高；手动设计注意力类型布局极为困难。我们提出DtR（Distill-then-Replace）方法：首先通过分块局部蒸馏，将预训练全注意力模块的权重迁移到对应的线性注意力模块；随后采用贪心层替换策略，迭代地用线性注意力块替代全注意力块，同时监测目标任务上的验证性能。DtR仅需一次高效遍历即可生成特定任务的混合模型，无需昂贵的重新训练或神经网络架构搜索，可应用于任何预训练的全注意力主干网络以支持多样下游任务。
原文摘要 · Abstract (English)
Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity with respect to sequence length limits practical deployment. Linear attention mechanisms offer linear or near-linear scaling yet often incur performance degradation. Hybrid models that integrate full and linear attention layers promise a balance between efficiency and expressiveness, but face two major challenges: training such hybrid models from scratch is computationally expensive, and manually designing the optimal placement of attention types is highly nontrivial. We propose DtR (Distill-then-Replace), which first transfers weights from the pretrained full-attention modules to its linear attention counterparts through blockwise local distillation, and then applies a greedy layer replacement strategy that iteratively substitutes full attention blocks with linear ones while monitoring validation performance on the target task. DtR yields a task-specific hybrid model in a single efficient pass, without costly re-training or neural architecture search, and can be applied to any pretrained full-attention backbone for diverse downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。