让视觉状态空间模型学会感知空间结构,提升图像分割精度。
OCTOPUS: Enhancing the Spatial-Awareness of Vision SSMs with Multi-Dimensional Scans and Traversal Selection
- 沿8个主方向双向扫描,实现多维度信息交互
- 在保持线性复杂度下,边界保真度显著提升
- 适合需要高效且精准分割的视觉任务
状态空间模型(SSMs)因线性复杂度在文本建模中表现优异,但在视觉任务中受限于因果结构,难以捕捉像素或补丁间的局部空间关系,常忽略相邻相关区域而连接非邻接补丁。为此,我们提出OCTOPUS,一种新型架构,在保留全局上下文的同时维持局部空间结构,并保持SSM的线性复杂度。OCTOPUS沿水平、垂直及对角线等八个主方向进行正向或反向离散递归,有效促进所有空间相连区域间的信息交换,同时保持无关补丁间的独立性。该设计实现多方向递归,以SSM级效率捕获全局与局部空间结构。在分类与分割基准测试中,OCTOPUS在边界保持和区域一致性方面表现显著优于现有V-SSM模型,同时保持相对更高的分类准确率。结果表明,OCTOPUS可作为构建空间感知且计算高效的视觉架构的基础方法。
原文摘要 · Abstract (English)
State space models (SSMs) have recently emerged as an alternative to transformers due to their unique ability of modeling global relationships in text with linear complexity. However, their success in vision tasks has been limited due to their causal formulation, which is suitable for sequential text but detrimental in the spatial domain where causality breaks the inherent spatial relationships among pixels or patches. As a result, standard SSMs fail to capture local spatial coherence, often linking non-adjacent patches while ignoring neighboring ones that are visually correlated. To address these limitations, we introduce OCTOPUS , a novel architecture that preserves both global context and local spatial structure within images, while maintaining the linear complexity of SSMs. OCTOPUS performs discrete reoccurrence along eight principal orientations, going forward or backward in the horizontal, vertical, and diagonal directions, allowing effective information exchange across all spatially connected regions while maintaining independence among unrelated patches. This design enables multi-directional recurrence, capturing both global context and local spatial structure with SSM-level efficiency. In our classification and segmentation benchmarks, OCTOPUS demonstrates notable improvements in boundary preservation and region consistency, as evident from the segmentation results, while maintaining relatively better classification accuracy compared to existing V-SSM based models. These results suggest that OCTOPUS appears as a foundation method for multi-directional recurrence as a scalable and effective mechanism for building spatially aware and computationally efficient vision architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。