发现CNN与Transformer共享同一学习机制，揭示注意力头的分工原理。
Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi
- 通过节点性能指标统一分析CNN与ViT的底层学习机制。
- 提出无需降损的节点对角剪枝法，实现模型压缩。
- 发现注意力头自发分工会聚，形成标签专家，适合模型可解释性研究。
卷积神经网络（CNN）捕捉输入图像中的短程相关性并逐层推进，而视觉变换器（ViT）架构通过多层全连接的变换器编码器评估长程相关性。两者虽设计目标均为复杂分类任务，但视角不同。本研究证明，CNN与ViT源于统一的底层学习机制，该机制定量测量前馈（FF）和多头注意力（MHA）子模块中每个节点的单节点性能（SNP）。每个节点识别一组可能的输出标签簇，额外噪声表现为簇外标签。这些特征在变换器编码器中逐步锐化，提升信噪比。该统一机制带来两项主要发现：其一，实现了高效的节点对角连接（ANDC）剪枝技术，不影响精度；其二，基于SNP，MHA头间发生自发对称性破缺，各头通过其节点性能协同聚焦于部分标签，成为指定标签的专家，体现定量的多头注意力生存模式。这一受统计力学启发的视角，使宏观网络行为可由微观节点性能推演。实验基于紧凑型卷积变换器在CIFAR-100和Flowers-102数据集上完成，呼吁推广至其他架构与自然语言处理等应用。
原文摘要 · Abstract (English)
Convolutional neural networks (CNNs) evaluate short-range correlations in input images which progress along the layers, whereas vision transformer (ViT) architectures evaluate long-range correlations, using repeated transformer encoders composed of fully connected layers. Both are designed to solve complex classification tasks but from different perspectives. This study demonstrates that CNNs and ViT architectures stem from a unified underlying learning mechanism, which quantitatively measures the single-nodal performance (SNP) of each node in feedforward (FF) and multi-head attention (MHA) sub-blocks. Each node identifies small clusters of possible output labels, with additional noise represented as labels outside these clusters. These features are progressively sharpened along the transformer encoders, enhancing the signal-to-noise ratio. This unified underlying learning mechanism leads to two main findings. First, it enables an efficient applied nodal diagonal connection (ANDC) pruning technique without affecting the accuracy. Second, based on the SNP, spontaneous symmetry breaking occurs among the MHA heads, such that each head focuses its attention on a subset of labels through cooperation among its SNPs. Consequently, each head becomes an expert in recognizing its designated labels, representing a quantitative MHA modus vivendi mechanism. This statistical mechanics inspired viewpoint enables to reveal macroscopic behavior of the entire network from the microscopic performance of each node. These results are based on a compact convolutional transformer architecture trained on the CIFAR-100 and Flowers-102 datasets and call for their extension to other architectures and applications, such as natural language processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。