用频谱分析增强ViT注意力机制，提升图像分类泛化能力
SIEFormer: Spectral-Interpretable and -Enhanced Transformer for Generalized Category Discovery
- 通过隐式与显式频谱分支，融合图拉普拉斯与傅里叶变换建模特征相关性
- 在多个数据集上达到最新最优性能，尤其在广义类别发现任务中显著领先
- 适合关注视觉变压器改进与跨类别识别的科研人员参考
本文提出一种新方法——频谱可解释且增强型变压器（SIEFormer），利用频谱分析重新诠释视觉变压器（ViT）中的注意力机制，并增强特征适应性，特别针对具有挑战性的广义类别发现（GCD）任务。SIEFormer由两个主分支组成，分别对应ViT的隐式与显式频谱视角，实现联合优化。隐式分支使用不同类型的图拉普拉斯矩阵建模标记间的局部结构相关性，并引入新型带通自适应滤波层（BaF），可灵活执行带通与带阻滤波。显式分支则引入可调节滤波层（MFL），通过对输入‘值’特征进行傅里叶变换，在频域中使用可学习参数调制变换信号，再通过逆傅里叶变换获得增强特征。大量实验表明，该方法在多个图像识别数据集上达到最先进水平，消融研究与可视化进一步验证了其有效性。
原文摘要 · Abstract (English)
This paper presents a novel approach, Spectral-Interpretable and -Enhanced Transformer (SIEFormer), which leverages spectral analysis to reinterpret the attention mechanism within Vision Transformer (ViT) and enhance feature adaptability, with particular emphasis on challenging Generalized Category Discovery (GCD) tasks. The proposed SIEFormer is composed of two main branches, each corresponding to an implicit and explicit spectral perspective of the ViT, enabling joint optimization. The implicit branch realizes the use of different types of graph Laplacians to model the local structure correlations of tokens, along with a novel Band-adaptive Filter (BaF) layer that can flexibly perform both band-pass and band-reject filtering. The explicit branch, on the other hand, introduces a Maneuverable Filtering Layer (MFL) that learns global dependencies among tokens by applying the Fourier transform to the input ``value" features, modulating the transformed signal with a set of learnable parameters in the frequency domain, and then performing an inverse Fourier transform to obtain the enhanced features. Extensive experiments reveal state-of-the-art performance on multiple image recognition datasets, reaffirming the superiority of our approach through ablation studies and visualizations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。