arXiv:2607.18625cs.CVcs.AI2026-07

对比两种视觉模型的编码方式,发现方向信息比幅度更重要。

Norm or Direction? Decoding Vision Mambas for High-Resolution Vision

论文配图:Norm or Direction? Decoding Vision Mambas for High-Resolution Vision
图 1 · 摘自论文原文
  • 拆解视觉特征的幅度与方向,揭示模型编码差异
  • VMamba靠方向保留语义,MambaOut依赖高幅度前景区域
  • 适合高分辨率图像分割任务的模型设计参考

Vision Mamba 用线性复杂度的选择性状态空间模型(SSMs)替代二次自注意力,成为高效的视觉主干网络。然而,MambaOut 表明门控卷积块可在图像分类上达到甚至超过 VMamba,引发对 SSM 必要性的质疑。我们通过交叉模型中心核对齐(CKA)分析发现,VMamba 最后阶段的块在表示层面显著区别于 MambaOut 及其前序块。聚焦于最后块特征,将每个空间标记分解为幅度和方向:MambaOut 将判别信息集中于与 Grad-CAM 对齐的高幅度前景标记;而 VMamba 的高幅度标记主要分布在背景区域,与 Grad-CAM 不一致,但判别信号主要保留在标记方向中。这表明两者采用不同编码策略。我们将其与高分辨率分类和语义分割关联:VMamba 在物体区域广泛分布逻辑支持,而 MambaOut 依赖稀疏主导标记,随着标记数量增加稳定性下降。全微调下,VMamba 在分割任务中持续优于 MambaOut。结果表明,VMamba 在密集预测中的优势不仅源于 SSM 机制或序列长度,更在于语义证据在标记幅度与方向上的组织方式。最终结论是,幅度与方向结构是提升视觉主干网络的关键轴线,尤其在密集监督场景下。

原文摘要 · Abstract (English)

Vision Mamba models replace quadratic self-attention with linear complexity selective state space models (SSMs), emerging as efficient visual backbones. However, MambaOut demonstrates that a Gated CNN block can match or exceed VMamba on image classification, questioning the necessity of SSMs for vision. This raises a fundamental question: do VMamba and MambaOut encode visual information differently at the representation level? To investigate, we apply cross model centered kernel alignment (CKA) analysis and find that VMamba's final stage blocks form representations distinctly different from both MambaOut and its own preceding blocks. We therefore focus on the final block features, decomposing each spatial token into magnitude and direction. MambaOut concentrates class-discriminative information in high-norm foreground tokens that align with Grad-CAM attribution. VMamba, by contrast, produces high-norm tokens predominantly in background regions, misaligned with Grad-CAM, yet preserves discriminative signals primarily in token directions. These observations reveal that the two models rely on different encoding strategies. We connect this difference to high-resolution classification and semantic segmentation. VMamba distributes logit support broadly across object regions, whereas MambaOut relies on sparse dominant tokens, a strategy that becomes less stable as token counts grow. Under full fine-tuning for segmentation, VMamba consistently outperforms MambaOut. These results suggest that VMamba's advantage in dense prediction stems not merely from the SSM mechanism or sequence length, but from how semantic evidence is organized across token magnitude, direction. Ultimately, we conclude that token magnitude and directional structure serve as critical axes for improving visual backbones, particularly under dense supervision.

视觉建模特征分解分割性能方向编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。