arXiv:2510.05901cs.LGcs.AI2025-10被引 3

发现混合线性注意力方法中线性组件被忽略,提出三种新方案确保其真正发挥作用。

Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods

  • 通过推理时混合线性与滑动窗口注意力,平衡组件使用。
  • 在不牺牲效率的前提下恢复原始模型90%以上性能。
  • 适合关注模型可解释性与高效推理的AI研究者。

Transformer 的二次计算复杂度限制了其可扩展性,尽管性能出色。虽然线性注意力将复杂度降至线性,但从头预训练此类模型在大多数情况下仍成本过高。近期的后训练线性化方法能高效将预训练 Transformer 转换为线性模型,常采用结合线性注意力与滑动窗口 softmax(SWA)的混合策略。我们发现一个关键缺陷:现有混合方法实际上绕过了线性组件,几乎完全依赖 SWA。组件级诊断显示,这种未被察觉的行为源于对常识基准测试的忽视性评估实践。为此我们提出三种解决方案:(i) 推理时将纯线性转换与滑动窗口 softmax 混合;(ii) HedgeCATs,结合注意力权重迁移与定向 LoRA 微调;(iii) 有计划的滑动窗口丢弃(SSD),在训练中随机抑制 softmax 分支以防止组件坍缩。所提方法在保持计算效率的同时恢复大部分基模型性能,并确保线性注意力被真实采用,重新确立了混合转换中性能归因的有效性。

原文摘要 · Abstract (English)

Transformers' quadratic computational complexity limits their scalability despite remarkable performance. While linear attention reduces this to linear complexity, pre-training such models from scratch remains, in most cases, prohibitively expensive. Recent post-training linearisation methods convert pre-trained Transformers to linear models efficiently, often using hybrid approaches that combine linear attention with sliding-window softmax. We identify a critical flaw: existing hybrid methods inadvertently bypass the linear component, relying almost entirely on SWA. Component-level diagnostics reveal this previously undetected behaviour stems from overlooked evaluation practices on common-sense benchmarks. We propose three solutions to ensure balanced component usage: (i) inference-time hybridisation of linear-only conversions with sliding-window softmax; (ii) HedgeCATs, combining attention-weight transfer with targeted LoRA fine-tuning; and (iii) Scheduled Sliding-window Dropout (SSD), which stochastically suppresses the softmax branch during training to prevent component collapse. Our methods maintain computational efficiency while recovering most base model performance and ensuring genuine linear attention adoption, restoring the validity of performance attributions in hybrid conversions.

线性注意力模型压缩Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。