让视觉Transformer动态调整通道特征，提升准确率并减少计算量
CAViT -- Channel-Aware Vision Transformer for Dynamic Feature Fusion
- 用双注意力机制替代固定MLP，实现内容感知的通道自适应融合
- 在5个数据集上最高提升3.6%准确率，参数与计算量降低超30%
- 适合需要高效高精度图像识别的场景，如医疗影像分析
视觉变压器（ViTs）通过自注意力建模长程空间关系，在多项计算机视觉任务中表现优异。然而，其通道混合机制仍为静态，依赖固定的多层感知机（MLPs），缺乏对输入内容的自适应能力。本文提出CAViT，一种双注意力架构，将静态MLP替换为基于注意力的动态特征交互机制。CAViT每个变换块先执行空间自注意力，再进行通道自注意力，使模型能根据全局图像上下文动态重校准特征表示。该统一且内容感知的令牌混合策略提升了表征表达力，同时不增加深度或复杂度。我们在涵盖自然与医学领域的五个基准数据集上验证了CAViT，结果表明其相比标准ViT基线最高提升3.6%准确率，参数量与浮点运算量均降低超过30%。定性注意力图显示激活模式更清晰、语义更明确，证实了注意力驱动令牌混合的有效性。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have demonstrated strong performance across a range of computer vision tasks by modeling long-range spatial interactions via self-attention. However, channel-wise mixing in ViTs remains static, relying on fixed multilayer perceptrons (MLPs) that lack adaptability to input content. We introduce 'CAViT', a dual-attention architecture that replaces the static MLP with a dynamic, attention-based mechanism for feature interaction. Each Transformer block in CAViT performs spatial self-attention followed by channel-wise self-attention, allowing the model to dynamically recalibrate feature representations based on global image context. This unified and content-aware token mixing strategy enhances representational expressiveness without increasing depth or complexity. We validate CAViT across five benchmark datasets spanning both natural and medical domains, where it outperforms the standard ViT baseline by up to +3.6% in accuracy, while reducing parameter count and FLOPs by over 30%. Qualitative attention maps reveal sharper and semantically meaningful activation patterns, validating the effectiveness of our attention-driven token mixing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。