arXiv:2511.09870cs.CV2025-11中稿 · AAAI被引 2

用深度信息引导查询,让SAM模型无须人工提示就能高效检测视频中的显著目标。

SAM-DAQ: Segment Anything Model with Depth-guided Adaptive Queries for RGB-D Video Salient Object Detection

  • 通过深度引导的并行适配器融合多模态特征,实现无需手动提示的图像编码。
  • 设计查询驱动的时序记忆模块,可同时提取帧间一致性并动态更新查询表示。
  • 在三个数据集上全面超越现有方法,尤其在内存和计算效率上优势明显。

近期,通用分割模型 SAM 引起广泛关注,被视为视觉基础模型。然而将其直接用于 RGB-D 视频显著目标检测(RGB-D VSOD)时面临三大挑战:依赖人工提示、序列适配器内存开销大、记忆注意力计算负担重。为此,本文提出一种新方法 SAM-DAQ,将 SAM2 适配至从视频中精准识别显著目标,通过统一框架融合深度与时间线索。首先,采用基于并行适配器的多模态图像编码器(PAMIE),在跳跃连接中集成多个深度引导并行适配器(DPAs),在无提示条件下微调冻结的 SAM 编码器,利用深度信息促进多模态特征融合。其次,引入查询驱动的时序记忆(QTM)模块,将记忆库与提示嵌入统一为可学习流程;通过同时使用帧级与视频级查询,不仅能选择性提取时序一致性特征,还可迭代更新查询的时序表示。在三个 RGB-D VSOD 数据集上的实验表明,SAM-DAQ 在所有评估指标上均持续优于现有最先进方法。

原文摘要 · Abstract (English)

Recently segment anything model (SAM) has attracted widespread concerns, and it is often treated as a vision foundation model for universal segmentation. Some researchers have attempted to directly apply the foundation model to the RGB-D video salient object detection (RGB-D VSOD) task, which often encounters three challenges, including the dependence on manual prompts, the high memory consumption of sequential adapters, and the computational burden of memory attention. To address the limitations, we propose a novel method, namely Segment Anything Model with Depth-guided Adaptive Queries (SAM-DAQ), which adapts SAM2 to pop-out salient objects from videos by seamlessly integrating depth and temporal cues within a unified framework. Firstly, we deploy a parallel adapter-based multi-modal image encoder (PAMIE), which incorporates several depth-guided parallel adapters (DPAs) in a skip-connection way. Remarkably, we fine-tune the frozen SAM encoder under prompt-free conditions, where the DPA utilizes depth cues to facilitate the fusion of multi-modal features. Secondly, we deploy a query-driven temporal memory (QTM) module, which unifies the memory bank and prompt embeddings into a learnable pipeline. Concretely, by leveraging both frame-level queries and video-level queries simultaneously, the QTM module can not only selectively extract temporal consistency features but also iteratively update the temporal representations of the queries. Extensive experiments are conducted on three RGB-D VSOD datasets, and the results show that the proposed SAM-DAQ consistently outperforms state-of-the-art methods in terms of all evaluation metrics.

视频显著检测深度引导SAM时序建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。