用近邻聚合统一卷积与注意力,揭示二者连续谱关系
Interpolation between Convolution and Attention via K-Nearest Neighbors

- 以最近邻聚合统一卷积与注意力,基于空间或特征相似性选邻域
- 框架可精确复现标准卷积、深度卷积及自注意力等模型
- 支持灵活配置,适合研究局部到全局聚合的中间路径
从卷积神经网络到Transformer的转变重塑了计算机视觉领域,但这两类架构通常被视为根本不同:卷积依赖空间局部操作,而Transformer基于全局自注意力。本文认为,尽管表观不同,卷积与自注意力均可统一于一个k-最近邻聚合框架中。关键洞见在于,两者均是邻域选择与加权聚合的特例:卷积按空间邻近选邻,自注意力按特征相似性选邻,表明它们处于连续谱而非类别迥异的计算模式。我们提出卷积最近邻(ConvNN)框架,精确复现标准卷积和深度卷积(通过归一化空间坐标限制邻域),以及自注意力及其稀疏变体(如KVT-attention,通过缩放点积相似性替换空间邻近)。除这些特例外,ConvNN可作为卷积与注意力层的即插即用替代品,通过可配置的相似度函数、邻域选择策略、位置编码和聚合核,系统探索局部与全局聚合之间的中间谱带。
原文摘要 · Abstract (English)
The shift from Convolutional Neural Networks to Transformers has reshaped computer vision, yet these two architectural families are typically viewed as fundamentally distinct. Convolutional Neural Networks are defined by spatially local convolution operations, while Transformers rely on global self-attention. We argue that convolution and self-attention, despite their apparent differences, can be unified within a single k-nearest neighbor aggregation framework. The critical insight is that both operations are special cases of neighbor selection and weighted aggregation. Convolution selects neighbors by spatial proximity while self-attention selects by feature similarity, revealing that they lie on a continuous spectrum rather than representing categorically different computations. We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that formalizes this connection. ConvNN exactly recovers standard and depthwise convolution by restricting neighbor selection to normalized spatial coordinates, and exactly recovers self-attention and its sparse variants, including KVT-attention, by replacing spatial proximity with scaled dot-product similarity. Beyond these special cases, ConvNN serves as a drop-in replacement for both convolution and attention layers, enabling systematic exploration of the intermediate spectrum between local and global aggregation through configurable similarity functions, neighbor selection strategies, positional encodings, and aggregation kernels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。