arXiv:2608.17475cs.CV2026-08

用可校准的频率适配器提升单流SAM在多模态显著目标检测中的精度与效率。

S$^3$AM: A Single-Stream SAM with Reliability-Calibrated Frequency Adapter for Multi-modal Salient Object Detection

论文配图:S$^3$AM: A Single-Stream SAM with Reliability-Calibrated Frequency Adapter for Multi-modal Salient Object Detection
图 1 · 摘自论文原文
  • 设计频率专家混合模块,通过小波变换融合多模态高频信息。
  • 提出双门控校准机制,动态控制噪声干扰下的特征注入强度。
  • 仅12.20M可训练参数,性能媲美双流模型,适合轻量部署。

视觉基础模型通过参数高效微调和提示学习推动了多模态显著目标检测(MSOD)的发展。然而,现有基于Segment Anything Model(SAM)的MSOD方法常依赖双流编码器或辅助提示生成器,导致计算冗余。尽管单流方案可降低开销,但早期融合可能将噪声或错位的高频辅助信息传递至主干网络。本文提出一种新型单流框架,将可靠性校准的频率适配集成至SAM主干中以实现MSOD。该方法避免重复使用基础主干,同时显式控制辅助频率注入。具体地,设计了频率专家混合模块,利用平稳小波变换分解各模态并聚合跨模态频率信息。进一步引入带有双门控校准机制的可靠性校准频率适配器,选择性传播校准后的残差,并联合调控其注入强度与跨模态可靠性。最后,通过超网络引导的语义-结构解码器,结合来自主干的语义掩码特征与基于Mamba的结构细节恢复。在RGB-D、RGB-T和RGB-NIR多模态显著目标检测基准上的全面实验表明,该框架仅需12.20M可训练参数(占总参数的5.4%),即可达到具有竞争力的性能。代码将开源于https://github.com/xuboyue1999/SSSAM。

原文摘要 · Abstract (English)

Vision foundation models have recently advanced multi-modal salient object detection (MSOD) through parameter-efficient tuning and prompt learning. However, existing Segment Anything Model (SAM)-adapted MSOD methods often rely on dual-stream encoders or auxiliary prompt generators, leading to redundant computation. Although a single-stream alternative can reduce this cost, early fusion may also propagate noisy or misaligned auxiliary high-frequency cues through the backbone. In this paper, we propose a novel single-stream framework that integrates reliability-calibrated frequency adaptation into the adopted SAM backbone for MSOD. It avoids duplicated foundation backbones while explicitly controlling auxiliary frequency injection. Specifically, we design a mixture of frequency experts module, which uses the stationary wavelet transform to decompose each modality and aggregate cross-modal frequency information. We further introduce a reliability-calibrated frequency adapter with a dual-gate calibration mechanism, which selectively propagates the calibrated residual across transformer stages while jointly controlling its injection strength and cross-modal reliability. A hypernetwork-guided semantic-structural decoder then combines semantic mask features from the adopted backbone with Mamba-based structural detail recovery. Comprehensive experiments on RGB-D, RGB-T, and RGB-NIR salient object detection benchmarks validate that the proposed framework achieves competitive performance with only 12.20M trainable parameters, accounting for 5.4\% of the total parameters. The code will be available at https://github.com/xuboyue1999/SSSAM.

多模态显著目标SAM轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。