arXiv:2511.14137cs.CV2025-11被引 1

统一卷积与注意力机制,揭示二者本质同源。

Unifying Convolution and Attention via Convolutional Nearest Neighbors

  • 用最近邻聚合框架统一卷积与注意力,通过相似性选择邻居
  • 在ImageNet上提升3.0%准确率,超越标准自注意力0.7%
  • 适合想融合局部与全局特征的视觉模型设计者

卷积神经网络与视觉变换器是计算机视觉中的两大主流架构,分别基于空间局部卷积和全局自注意力。尽管表象不同,我们发现两者均为同一k-最近邻聚合框架的特例:卷积按空间距离选邻域,注意力按特征相似性选邻域,二者处于同一操作谱的两端。我们提出卷积最近邻(ConvNN)框架,可精确复现标准卷积、深度卷积、自注意力及稀疏注意力(如KVT-attention)等变体,并通过可配置的相似性函数、位置编码和聚合核,揭示二者间的邻居选择策略设计空间。在ImageNet-1K分类任务中,我们在ResNet-50的混合分支结构中引入ConvNN,使准确率提升3.0%;在ViT-Base中使用ConvNN-attention达到81.64%的top-1准确率,比标准多头自注意力高0.7%。结果表明,ConvNN为连接卷积与注意力计算提供了原则性基础。

原文摘要 · Abstract (English)

Convolutional Neural Networks and Vision Transformers are the two dominant architectural families in computer vision, defined by spatially local convolution and global self-attention respectively. Despite their apparent differences, we show that both operations are special cases of a single $k$-nearest neighbor aggregation framework: convolution selects neighbors by spatial proximity while attention selects by feature similarity, placing them at two ends of a shared operational spectrum. We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that exactly recovers standard and depthwise convolution, self-attention, and sparse attention variants including KVT-attention as special cases, and exposes the design space of neighbor-selection strategies between them through configurable similarity functions, positional encodings, and aggregation kernels. We validate ConvNN on ImageNet-1K classification across two complementary architectures: a hybrid branching layer in ResNet-50 that combines local and global feature learning, improving top-1 accuracy by 3.0% over the ResNet-50 baseline, and ConvNN-attention in ViT-Base that achieves 81.64% top-1 accuracy, surpassing standard multi-head self-attention by 0.7%. Together, these results demonstrate that ConvNN provides a principled foundation for designing operations that bridge convolutional and attention-based computation.

卷积注意力统一框架视觉模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。