提出可自适应融合空间结构的视觉状态空间模型,提升图像与3D感知性能。
Deformba: Vision State Space Model with Adaptive State Fusion

- 动态调整图像块的空间结构信息,替代固定扫描方式
- 在图像分类、检测、分割及BEV感知任务中表现优异
- 支持多模态融合,适用于需跨流交互的复杂感知任务
状态空间模型(SSMs)作为Transformer的高效替代方案,具备线性时间复杂度和出色的序列建模能力。然而其在视觉任务中的应用仍面临挑战:现有视觉SSM多依赖人工设计的固定扫描方式将图像块展平为序列,强加预定义几何结构并增加复杂性;且由于其固有的因果性和自引用特性,难以支持不同信息流间的查询交互,限制了在多视角3D融合等关键感知任务中的应用。为此,本文提出Deformba,一种上下文自适应方法,可在保持SSM线性复杂度的同时动态增强空间结构信息,并支持类似交叉注意力的多模态融合。通过在图像分类、目标检测、分割以及鸟瞰图(BEV)感知等2D与3D视觉任务上进行广泛实验,结果表明Deformba在多个视觉感知基准上均达到强劲性能。
原文摘要 · Abstract (English)
State Space Models (SSMs) have emerged as a powerful and efficient alternative to Transformers, demonstrating linear-time complexity and exceptional sequence modeling capabilities. However, their application to vision tasks remains challenging. First, existing vision SSMs largely depend on manually designed fixed scanning methods to flatten image patches into sequences, which imposes predefined geometric structures and increases the complexity. Second, the broader adoption of vision SSMs is hindered in domains that require query-based interactions between distinct information streams. This is a result of the inherently causal and self-referential nature of SSMs designed for 1D sequence modeling tasks. This fusion mechanism is indispensable for critical perception tasks such as multi-view 3D fusion. To address these limitations, we propose Deformba, a context adaptive method that dynamically augments the spatial structural information while maintaining the linear complexity of SSMs. Deformba also allows multi-modal fusion like cross attention. To demonstrate the effectiveness and general applicability of Deformba, we test its performance on general 2D vision tasks such as image classification, object detection, and segmentation, as well as 3D vision tasks like BEV perception. Extensive experiments show that Deformba achieves strong performance across various visual perception benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。