用动态参数替代注意力，实现线性复杂度的全局建模
Linear-Time Global Visual Modeling without Explicit Attention
- 将注意力重构成带动态参数的MLP，隐式捕捉全局上下文
- 在视觉任务中达到Transformer水平性能，计算复杂度降至线性
- 适合追求高效序列建模的开发者，尤其关注推理速度的场景
现有研究普遍认为Transformer的全局序列建模能力依赖于显式的注意力权重计算，该过程固有地带来二次方复杂度。本文提出新视角：注意力可数学重构为带有动态预测参数的多层感知机（MLP）。通过此视角，我们解释注意力的全局建模能力并非来自显式的令牌聚合，而是由动态生成的参数作为全局上下文的压缩表示所驱动。受此启发，我们探讨一个根本问题：能否完全通过动态参数化实现Transformer级别的序列全局建模，同时保持线性复杂度，从而彻底取代显式注意力？为此，我们设计了多种动态参数预测策略，并将其集成至标准网络层中。在视觉模型上的大量实验证明，动态参数化确实可作为高效且线性复杂度的显式注意力替代方案，为高效序列建模开辟新路径。代码已开源：https://github.com/LeapLabTHU/WeightFormer。
原文摘要 · Abstract (English)
Existing research largely attributes the global sequence modeling capability of Transformers to the explicit computation of attention weights, a process that inherently incurs quadratic computational complexity. In this work, we offer a novel perspective: we demonstrate that attention can be mathematically reframed as a Multi-Layer Perceptron (MLP) equipped with dynamically predicted parameters. Through this lens, we explain attention's global modeling power not as explicit token-wise aggregation, but as an implicit process where dynamically generated parameters act as a compressed representation of the global context. Inspired by this insight, we investigate a fundamental question: can we achieve Transformer-level sequence global modeling entirely through dynamic parameterization while maintaining linear complexity, effectively replacing explicit attention? To explore this, we design various dynamic parameter prediction strategies and integrate them into standard network layers. Extensive empirical studies on vision models demonstrate that dynamic parameterization can indeed serve as a highly effective, linear-complexity alternative to explicit attention, opening new pathways for efficient sequence modeling. Code is available at https://github.com/LeapLabTHU/WeightFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。