通过分层混合线性与softmax注意力，提升视觉模型精度与效率
SoLA-Vision: Fine-grained Layer-wise Linear Softmax Hybrid Attention
- 按层精细混合线性与softmax注意力，灵活控制计算方式
- 在ImageNet-1K上超越纯线性与其它混合模型，密度预测任务显著领先
- 仅需少量全局softmax层即可实现高精度，适合高分辨率视觉应用
标准softmax自注意力在视觉任务中表现优异，但复杂度高达O(N²)，限制了高分辨率部署。线性注意力将复杂度降至O(N)，但压缩状态表示可能削弱建模能力与精度。本文从层堆叠视角分析线性与softmax注意力在视觉表征学习中的差异，并系统实验了层间混合模式。结果表明，相比固定块内混合设计，细粒度层间混合可在更少softmax层下达到或超过性能。基于此，我们提出SoLA-Vision（Softmax-Linear Attention Vision），一种可灵活控制线性与softmax注意力融合的骨干网络。通过战略性插入少量全局softmax层，该模型在准确率与计算成本间取得良好平衡。在ImageNet-1K上优于纯线性及其它混合注意力模型；在密集预测任务中持续大幅超越强基线。代码将开源。
原文摘要 · Abstract (English)
Standard softmax self-attention excels in vision tasks but incurs quadratic complexity O(N^2), limiting high-resolution deployment. Linear attention reduces the cost to O(N), yet its compressed state representations can impair modeling capacity and accuracy. We present an analytical study that contrasts linear and softmax attention for visual representation learning from a layer-stacking perspective. We further conduct systematic experiments on layer-wise hybridization patterns of linear and softmax attention. Our results show that, compared with rigid intra-block hybrid designs, fine-grained layer-wise hybridization can match or surpass performance while requiring fewer softmax layers. Building on these findings, we propose SoLA-Vision (Softmax-Linear Attention Vision), a flexible layer-wise hybrid attention backbone that enables fine-grained control over how linear and softmax attention are integrated. By strategically inserting a small number of global softmax layers, SoLA-Vision achieves a strong trade-off between accuracy and computational cost. On ImageNet-1K, SoLA-Vision outperforms purely linear and other hybrid attention models. On dense prediction tasks, it consistently surpasses strong baselines by a considerable margin. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。