用图结构动态路由视觉特征,提升Mamba类模型的感知能力
Can Graphs Help Vision SSMs See Better?

- 为视觉状态空间模型设计图驱动的动态扫描机制
- 在多个任务上达到SOTA性能,计算开销小
- 可解释的语义位移场,揭示动态扫描本质
视觉状态空间模型继承了Mamba式选择性扫描的高效性和长程建模能力,但其性能高度依赖于二维视觉特征转换为一维标记序列的方式。现有扫描算子从预定义几何遍历到基于坐标的动态采样器不等,主要调整路径或采样位置,而非显式建模局部块间信息交换。为此,本文提出GraphScan:一种图诱导的动态扫描算子。每个标记构建空间受限的局部图,学习特征条件下的亲和度并带相对位置偏置,通过一步消息传递生成输出标记。处理前确保标记局部语义对齐,再经选择性状态空间模型进行全局聚合。GraphScan保持标记数量和图像尺寸线性扩展,将坐标条件插值替换为特征条件语义路由。集成至分层骨干网络后,GraphScan-Mamba在图像分类、目标检测、实例分割和语义分割任务上均达到视觉状态空间模型的最新水平,计算开销微小。分析显示,GraphScan在标记网格上产生可解释的位移场,提供动态扫描的语义与空间基础视图。结果表明,未来视觉状态空间模型应将扫描视为全局状态空间建模前的可学习局部语义路由。
原文摘要 · Abstract (English)
Vision state space models inherit the efficiency and long-range modeling ability of Mamba-style selective scans. However, their performance depends critically on the representation of two-dimensional visual features as one-dimensional token sequences. Existing scan operators range from predefined geometric traversals to dynamic coordinate-based samplers that reroute tokens through predicted offsets and interpolation. While effective, these mechanisms primarily adapt paths or sampling locations, rather than explicitly modeling which local patches should exchange information before global state-space mixing. This motivates a simple question: \emph{can graphs help vision state space models see better?} We introduce \textbf{GraphScan}, a graph-induced dynamic scanning operator for Vision SSMs. For each token, GraphScan constructs a spatially bounded local graph, learns feature-conditioned affinities with relative positional bias, and produces the output token by one-step message passing over its semantic neighborhood. The resulting tokens are locally grounded before being processed by the selective SSM for global aggregation. GraphScan preserves token count and linear scaling in image size, while replacing coordinate-conditioned interpolation with feature-conditioned semantic routing. Integrated into a hierarchical backbone, \textbf{GraphScan-Mamba} achieves state-of-the-art performance among Vision SSMs across image classification, object detection, instance segmentation, and semantic segmentation, with modest computational overhead. Our analysis further shows that GraphScan induces interpretable displacement fields over the token lattice, providing a semantic and spatially grounded view of dynamic scanning. These results suggest that future Vision SSMs should treat scanning not merely as geometric serialization, but as learned local semantic routing before global state-space modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。