arXiv:2605.11840cs.CV2026-05被引 1

用雷达调控Mamba的扫描过程,实现更准更快的多模态深度估计。

Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation

论文配图:Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation
图 1 · 摘自论文原文
  • 雷达直接干预Mamba的逐步扫描机制,通过调节步长和读出参数注入信息。
  • 在nuScenes数据集上,0-50米范围深度误差降低34.0%,单帧延迟仅26.8毫秒。
  • 方法天然支持雷达缺失时退化为纯图像模型,适合实际部署场景。

雷达-相机深度估计需将稀疏、全天候、有度量的雷达信号转化为密集像素级深度图。现有方法——拼接、置信度门控、稀疏监督、图结构提取——均在主干网络序列算子外融合雷达与图像特征,即使跨模态Mamba变体也未改变选择机制本身的单模态本质。本文认为,选择机制才是雷达介入的正确位置。提出雷达调制选择(RMS),一种极简且原理清晰的方法,将雷达注入Mamba的可选扫描:雷达在内部对步长Δ和读出C施加零初始化扰动,同时保留输入投影B与状态动力学A仅由图像决定。该设计在初始化时等价于预训练的纯图像Mamba,确保雷达仅在能提升精度处发挥作用。由此衍生出两个超越传统融合的优势:每一步递归中线性开销的跨模态耦合,以及雷达缺失时自然退化为图像主干的鲁棒性。将RMS集成至多视角扫描金字塔(MVSP)中,其融合算子与雷达在各尺度的空间覆盖相匹配。SemoDepth在nuScenes上达当前最优,0–50米、0–70米、0–80米范围的平均绝对误差分别降低34.0%、29.9%、29.9%;同时达到最低单帧延迟(26.8ms)。消融实验表明,扫描外特征融合无法在RMS基础上进一步提升精度,实证验证了扫描内选择可替代扫描外融合。

原文摘要 · Abstract (English)

Radar-camera depth estimation must turn an ultra-sparse, all-weather, metric radar signal into a dense per-pixel depth map. Existing methods -- concatenation, confidence-aware gating, sparse supervision, graph-based extraction -- combine radar and image features outside the backbone's sequence operator, and even cross-modal Mamba variants leave the selection mechanism itself unimodal. We argue that the selection mechanism is the right place for radar to enter. We introduce Radar-Modulated Selection (RMS), a minimal and principled way to inject radar into Mamba's selective scan: radar modulates the scan from within, adding zero-initialised perturbations to the step size $Δ$ and readout $\mathbf{C}$ while leaving the input projection $\mathbf{B}$ and state dynamics $\mathbf{A}$ image-only. The construction is exactly equivalent to a pretrained image-only Mamba at initialisation, ensuring radar only influences the model where it improves accuracy. Two further properties follow that out-of-scan fusion cannot offer: linear-cost cross-modal coupling at every recurrence step, and a natural fallback to the image-only backbone when radar is absent. We deploy RMS in a Multi-View Scan Pyramid (MVSP) that matches the fusion operator to radar's spatial reach at each scale. SemoDepth achieves state-of-the-art performance on nuScenes, reducing MAE by 34.0%, 29.9%, and 29.9% over the previous best at 0--50, 0--70, and 0--80m, while attaining the lowest single-frame latency (26.8ms). A further ablation shows that out-of-scan feature blending adds no accuracy on top of RMS, providing empirical validation that in-scan selection can replace out-of-scan fusion.

深度估计雷达融合Mamba实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。