arXiv:2512.00363cs.CV2025-12

轻量级多模态检测框架,用Mamba提升融合效率与精度。

MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters

  • 基于Mamba的双粒度融合,线性复杂度实现跨模态建模。
  • 引入频域感知适配器,在共享主干中高效提取模态特异性特征。
  • 适用于遥感目标检测,适合资源受限场景部署。

多模态遥感目标检测旨在通过融合不同模态的互补信息,在复杂条件下实现更准确、鲁棒的感知。然而,现有方法依赖注意力或可变形卷积融合模块,在性能与轻量化设计间难以平衡。共享主干提取特征导致模态特异性不足,而双流架构参数量几乎翻倍,限制实际部署。为此,我们提出MM-DETR,一种轻量高效的多模态目标检测框架。首先,设计基于Mamba的双粒度融合编码器,将全局交互重构为通道级动态门控,并利用一维选择性扫描实现线性复杂度的跨模态建模。进一步,将多模态融合重定义为模态补全问题,引入区域感知的二维选择性扫描补全分支,沿双向金字塔路径支持细粒度融合且开销极小。为减少参数冗余并保留强特征提取能力,我们在共享主干中插入轻量级频域感知模态适配器,采用空间-频率联合专家结构捕捉模态特异性线索,像素级路由动态调节专家贡献,实现高效的空间-频率融合。在四个多模态基准数据集上的大量实验验证了该方法的有效性与泛化能力。

原文摘要 · Abstract (English)

Multimodal remote sensing object detection aims to achieve more accurate and robust perception under challenging conditions by fusing complementary information from different modalities. However, existing approaches that rely on attention-based or deformable convolution fusion blocks still struggle to balance performance and lightweight design. Beyond fusion complexity, extracting modality features with shared backbones yields suboptimal representations due to insufficient modality-specific modeling, whereas dual-stream architectures nearly double the parameter count, ultimately limiting practical deployment. To this end, we propose MM-DETR, a lightweight and efficient framework for multimodal object detection. Specifically, we propose a Mamba-based dual granularity fusion encoder that reformulates global interaction as channel-wise dynamic gating and leverages a 1D selective scan for efficient cross-modal modeling with linear complexity. Following this design, we further reinterpret multimodal fusion as a modality completion problem. A region-aware 2D selective scanning completion branch is introduced to recover modality-specific cues, supporting fine-grained fusion along a bidirectional pyramid pathway with minimal overhead. To further reduce parameter redundancy while retaining strong feature extraction capability, a lightweight frequency-aware modality adapter is inserted into the shared backbone. This adapter employs a spatial-frequency co-expert structure to capture modality-specific cues, while a pixel-wise router dynamically balances expert contributions for efficient spatial-frequency fusion. Extensive experiments conducted on four multimodal benchmark datasets demonstrate the effectiveness and generalization capability of the proposed method.

多模态检测Mamba轻量化遥感

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。