通过通道采样置换提升多头注意力效率，参数更少、速度更快。
Towards Better Multi-head Attention via Channel-wise Sample Permutation
- 按通道对样本进行循环移位和分组排序，构造新型结构化注意力
- 在图像分类与长序列任务中性能相当或更好，参数减少30%以上
- 适合追求轻量化部署的视觉与自然语言模型应用
Transformer 在计算机视觉（如 ViT）和自然语言处理（如 BERT、GPT）中占据核心地位，其有效性主要归因于多头注意力（MHA）机制。本文提出一种简单新颖的通道采样置换（CSP）算子，实现参数更少、复杂度更低的结构化 MHA。给定输入矩阵，CSP 对不同通道的样本进行不同步长的循环移位，并对每通道分组样本进行排序。该操作等价于隐式实现跨通道注意力映射为置换矩阵，具备线性复杂度，能抑制数据表示中的秩坍缩风险。将代表性模型中的 MHA 替换为 CSP 后，在图像分类和长序列分析等任务上测试，结果表明：基于 CSP 的模型在参数量更少、计算成本更低的前提下，性能达到或超过经典 Transformer 及其先进变体。代码已公开于 https://github.com/DaShenZi721/CSP。
原文摘要 · Abstract (English)
Transformer plays a central role in many fundamental deep learning models, e.g., the ViT in computer vision and the BERT and GPT in natural language processing, whose effectiveness is mainly attributed to its multi-head attention (MHA) mechanism. In this study, we propose a simple and novel channel-wise sample permutation (CSP) operator, achieving a new structured MHA with fewer parameters and lower complexity. Given an input matrix, CSP circularly shifts the samples of different channels with various steps and then sorts grouped samples of each channel. This operator is equivalent to implicitly implementing cross-channel attention maps as permutation matrices, which achieves linear complexity and suppresses the risk of rank collapse when representing data. We replace the MHA of some representative models with CSP and test the CSP-based models in several discriminative tasks, including image classification and long sequence analysis. Experiments show that the CSP-based models achieve comparable or better performance with fewer parameters and lower computational costs than the classic Transformer and its state-of-the-art variants. The code is available at https://github.com/DaShenZi721/CSP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。