提出MirrorLA，让线性注意力更准，通过几何旋转保留负值信息。
MirrorLA: Reflecting Feature Map for Vision Linear Attention
- 用可学习的镜像反射替代传统截断，主动重定向特征分布
- 在多个基准上达到线性注意力最优性能，超越现有方法
- 适合追求高效高精度视觉模型的研究者与工程师
线性注意力将Transformer计算复杂度从二次降低到线性，但性能始终落后于Softmax注意力。我们发现根本原因在于核特征图的非负性约束：标准投影如ReLU作为‘被动截断’操作，无差别丢弃负域中的语义信息。为此提出MirrorLA，一种几何框架，以主动重定向替代被动截断。通过可学习的Householder反射，将特征几何旋转至非负象限，最大化信息保留。该方法通过统一的多尺度设计恢复表征密度：首先利用块内等距变换优化局部判别性，再通过方差感知调制稳定长程动态，最后通过跨头反射整合分散子空间，实现全局协方差混合。MirrorLA在标准基准上取得当前最佳性能，证明严格线性效率下仍可保持表征保真度。
原文摘要 · Abstract (English)
Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance. We identify the root cause of this degradation as the non-negativity constraint imposed on kernel feature maps: standard projections like ReLU act as "passive truncation" operators, indiscriminately discarding semantic information residing in the negative domain. We propose MirrorLA, a geometric framework that substitutes passive truncation with active reorientation. By leveraging learnable Householder reflections, MirrorLA rotates the feature geometry into the non-negative orthant to maximize information retention. Our approach restores representational density through a cohesive, multi-scale design: it first optimizes local discriminability via block-wise isometries, stabilizes long-context dynamics using variance-aware modulation to diversify activations, and finally, integrates dispersed subspaces via cross-head reflections to induce global covariance mixing. MirrorLA achieves state-of-the-art performance across standard benchmarks, demonstrating that strictly linear efficiency can be achieved without compromising representational fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。