arXiv:2510.20092cs.CV2025-10被引 1

用新卷积结构融合注意力机制优势,效率更高且性能更强

Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiency

  • 重新理解注意力优势,提出自适应路由与侧抑制机制
  • 仅用3×3卷积核实现超越多种注意力模型的性能
  • 适合追求高效高精度视觉模型的研究者和开发者

自注意力(SA)因其强大表达能力成为现代视觉主干网络的核心,但其二次复杂度仍是实际应用的关键瓶颈。传统卷积(Conv)虽具线性复杂度和强视觉先验,但现代改进仍难以弥补性能差距。本文通过追问SA为何优于Conv,发现两个关键机制:(1)自适应路由——注意力根据语义动态调节位置信息流,而卷积使用固定核;(2)侧抑制——注意力通过得分竞争抑制冗余,增强表示,而卷积缺乏此类抑制机制。基于此,提出有原则的卷积重构方法「有注意力卷积」(ATConv),内嵌上述机制。实验表明,仅用3×3核的ATConv在基础视觉任务中持续优于多种注意力结构。在此基础上构建的AttNet系列,在仅2700万参数下达到ImageNet-1K 84.4% Top-1准确率。在扩散图像生成中,将SiT-XL/2中的所有自注意力替换为3×3的ATConv,40万步内将ImageNet FID降低0.15且采样更快。代码已开源。

原文摘要 · Abstract (English)

Self-attention (SA) has become the cornerstone of modern vision backbones for its powerful expressivity over traditional Convolutions (Conv). However, its quadratic complexity remains a critical bottleneck for practical applications. Given that Conv offers linear complexity and strong visual priors, continuing efforts have been made to promote the renaissance of Conv. However, a persistent performance chasm remains, highlighting that these modernizations have not yet captured the intrinsic expressivity that defines SA. In this paper, we re-examine the design of the CNNs, directed by a key question: what principles give SA its edge over Conv? As a result, we reveal two fundamental insights that challenge the long-standing design intuitions in prior research (e.g., Receptive field). The two findings are: (1) \textit{Adaptive routing}: SA dynamically regulates positional information flow according to semantic content, whereas Conv employs static kernels uniformly across all positions. (2) \textit{Lateral inhibition}: SA induces score competition among token weighting, effectively suppressing redundancy and sharpening representations, whereas Conv filters lack such inhibitory dynamics and exhibit considerable redundancy. Based on this, we propose \textit{Attentive Convolution} (ATConv), a principled reformulation of the convolutional operator that intrinsically injects these principles. Interestingly, with only $3\times3$ kernels, ATConv consistently outperforms various SA mechanisms in fundamental vision tasks. Building on ATConv, we introduce AttNet, a CNN family that can attain \textbf{84.4\%} ImageNet-1K Top-1 accuracy with only 27M parameters. In diffusion-based image generation, replacing all SA with the proposed $3\times 3$ ATConv in SiT-XL/2 reduces ImageNet FID by 0.15 in 400k steps with faster sampling. Code is available at: github.com/price112/Attentive-Convolution.

卷积网络注意力机制模型效率视觉主干

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。