提出统一自注意力与卷积的新型操作，实现自适应与相对建模。
Translution: Unifying Self-attention and Convolution for Adaptive and Relative Modeling
- 融合自注意力的自适应选择与卷积的相对编码优势
- 在视觉与语言任务中优于传统自注意力模型
- 设计轻量版本α-Translution，降低参数开销
在建模数据时，需关注两个关键方面：一是识别与中心元素相关的部分（如卷积的感受野或自注意力的查询对象），二是有效编码这些元素。自注意力能自适应地选择相关元素，但依赖绝对位置嵌入进行结构学习；卷积以相对方式编码，但固定核大小限制了自适应选择能力。本文提出Translution，融合自注意力的自适应识别与卷积的相对编码优势。然而该融合导致参数量显著增加，超出现有计算资源承载能力。为此，我们提出轻量版α-Translution。在计算机视觉与自然语言处理任务上的实验表明，Translution（含α-Translution）在准确率上优于自注意力。代码已开源：https://github.com/hehefan/Translution。
原文摘要 · Abstract (English)
When modeling a given type of data, we consider it to involve two key aspects: 1) identifying relevant elements (e.g., image pixels or textual words) to a central element, as in a convolutional receptive field, or to a query element, as in self-attention, and 2) encoding these tokens effectively. Self-attention can adaptively identify these elements but relies on absolute positional embedding for structural representation learning. In contrast, convolution encodes elements in a relative manner, yet their fixed kernel size limits their ability to adaptively select the relevant elements. In this paper, we introduce Translution, an operation that unifies the adaptive identification capability of self-attention and the relative encoding advantage of convolution. However, this integration leads to a substantial increase in the number of parameters, exceeding most currently available computational resources. Therefore, we propose a lightweight variant of Translution, named α-Translution. Experiments on computer vision and natural language processing tasks show that Translution (including α-Translution) achieves superior accuracy compared to self-attention. The code is available at https://github.com/hehefan/Translution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。