稀疏注意力中学习路由比想象难,随机门限竟常胜过训练出的门控。
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
- 用可微软门控或硬Top-k门控,结果几乎等同于随机门控
- 学出来的门控在部署时性能暴跌,对比基准差了10倍以上
- 模型参数共适应导致路由信号被吸收,适合做后处理稀疏化
Transformer能否在训练中学会识别重要注意力项?理论上可以:注意力分布高度集中,小型门控网络可近乎完美地事后识别关键条目。但实践中几乎不行。当稀疏注意力端到端训练时,查询/键/值投影会与施加的掩码共同适应,使路由信号被吸收,导致学习到的门控表现仅略优于冻结的随机门控。我们称此为路由吸收,并在3100万参数的受控Transformer中提出四条独立证据:(1) 可微软门控无论是否学习,困惑度均接近(48.73±0.60 vs. 49.83±0.04,3个种子);(2) 硬Top-k门控通过掩码梯度为零;(3) 在共适应的Q/K/V上蒸馏的门控对理想掩码F1高,但部署到无掩码的Q/K/V时困惑度飙升至601.6(基准为48.6);(4) 训练中随机化掩码也无法阻止共适应(部署稠密为78.2,基线为37.3)。路由吸收与混合专家(MoE)中的类似现象相关,但注意力因共享参数存在跨层补偿路径,结构上更严重。结论是:端到端稀疏注意力中,门控与模型间参数不对称越大,吸收压力越强;而事后稀疏化方法可完全规避此问题。
原文摘要 · Abstract (English)
Can a transformer learn which attention entries matter during training? In principle, yes: attention distributions are highly concentrated, and a small gate network can identify the important entries post-hoc with near-perfect accuracy. In practice, barely. When sparse attention is trained end-to-end, the model's Q/K/V projections co-adapt to whatever mask is imposed, absorbing the routing signal until learned gates perform little better than frozen random gates. We call this routing absorption and present four independent lines of evidence for it in a controlled 31M-parameter transformer: (1) differentiable soft gating converges to nearly the same perplexity whether the gate is learned or random (48.73 +/- 0.60 vs. 49.83 +/- 0.04 over 3 seeds); (2) hard top-k gating receives exactly zero gradient through the mask; (3) a gate distilled onto co-adapted Q/K/V achieves high F1 against oracle masks but catastrophic perplexity when deployed (601.6 vs. 48.6 on mask-agnostic Q/K/V); and (4) stochastic mask randomization during training fails to prevent co-adaptation (78.2 ppl deployed dense vs. 37.3 baseline). We connect routing absorption to the same phenomenon in Mixture-of-Experts, where random routing matches learned routing because experts co-adapt to any router, but show that attention exhibits a structurally more severe form: shared Q/K/V parameters enable cross-layer compensation pathways absent in MoE, where experts are self-contained modules. The implication is that end-to-end sparse attention methods employing per-query token-level gating face absorption pressure proportional to the parameter asymmetry between the gate and the model, and that post-hoc approaches, which decouple representation learning from sparsification, sidestep this entirely.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。