提出谱式状态空间模型，让视觉模型旋转不变且更懂图像全局关系。
Spectral State Space Model for Rotation-Invariant Visual Representation Learning
- 用图拉普拉斯谱分解捕捉图像块间全局关系，不依赖原始方向。
- 在多个分类任务上超越主流视觉SSM模型，旋转后性能稳定。
- 适合需要旋转鲁棒性的视觉识别场景，如遥感或医学影像分析。
状态空间模型（SSMs）因其线性复杂度建模全局关系的能力，正成为视觉变压器（ViTs）的替代方案。然而，现有视觉SSM主要关注图像块的空间邻近关系，难以发现语义相关但不相邻的块之间的联系。这一局限源于图像数据缺乏固有的方向性。此外，当前视觉SSM对旋转敏感：其预设的扫描顺序依赖于原始图像方向，旋转后会导致处理序列不一致。为此，我们提出谱式VMamba（Spectral VMamba），通过图像块的图拉普拉斯谱分解，提取与方向无关的全局结构信息。结合旋转特征归一化模块（RFN），实现旋转不变性。实验表明，该模型在分类任务上优于领先的视觉SSM（如VMamba），同时保持相似的运行效率。
原文摘要 · Abstract (English)
State Space Models (SSMs) have recently emerged as an alternative to Vision Transformers (ViTs) due to their unique ability of modeling global relationships with linear complexity. SSMs are specifically designed to capture spatially proximate relationships of image patches. However, they fail to identify relationships between conceptually related yet not adjacent patches. This limitation arises from the non-causal nature of image data, which lacks inherent directional relationships. Additionally, current vision-based SSMs are highly sensitive to transformations such as rotation. Their predefined scanning directions depend on the original image orientation, which can cause the model to produce inconsistent patch-processing sequences after rotation. To address these limitations, we introduce Spectral VMamba, a novel approach that effectively captures the global structure within an image by leveraging spectral information derived from the graph Laplacian of image patches. Through spectral decomposition, our approach encodes patch relationships independently of image orientation, achieving rotation invariance with the aid of our Rotational Feature Normalizer (RFN) module. Our experiments on classification tasks show that Spectral VMamba outperforms the leading SSM models in vision, such as VMamba, while maintaining invariance to rotations and a providing a similar runtime efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。