剖析视觉Mamba模型中图像块的注意力分布规律
Visual Attention Exploration in Vision-Based Mamba Models
- 设计可视化工具分析图像块间的注意力模式
- 发现不同排列顺序显著影响注意力分布
- 适合研究视觉模型机制的开发者与学者
状态空间模型（SSMs）已成为Transformer的高效替代，具有线性复杂度，可更好扩展。Mamba作为最新进展，引入选择性扫描机制，为输入标记分配可训练权重，有效模拟注意力机制。该模型已通过将二维图像分解为小块并排成一维序列的方式成功拓展至视觉领域。然而，这些图像块在原始二维空间位置上的相互作用关系尚不明确，且序列排列顺序也显著影响注意力分布。为此，本文提出一种专为视觉Mamba模型设计的可视化分析工具，深入理解不同Mamba模块中注意力在图像块间的分布及其随模型演进的变化。利用该工具，还研究了不同图像块排列策略对学习到的注意力模式的影响，进一步揭示模型行为机制。
原文摘要 · Abstract (English)
State space models (SSMs) have emerged as an efficient alternative to transformer-based models, offering linear complexity that scales better than transformers. One of the latest advances in SSMs, Mamba, introduces a selective scan mechanism that assigns trainable weights to input tokens, effectively mimicking the attention mechanism. Mamba has also been successfully extended to the vision domain by decomposing 2D images into smaller patches and arranging them as 1D sequences. However, it remains unclear how these patches interact with (or attend to) each other in relation to their original 2D spatial location. Additionally, the order used to arrange the patches into a sequence also significantly impacts their attention distribution. To better understand the attention between patches and explore the attention patterns, we introduce a visual analytics tool specifically designed for vision-based Mamba models. This tool enables a deeper understanding of how attention is distributed across patches in different Mamba blocks and how it evolves throughout a Mamba model. Using the tool, we also investigate the impact of different patch-ordering strategies on the learned attention, offering further insights into the model's behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。