arXiv:2508.04418cs.MMcs.CV2025-08AAAI被引 11

让AI先理解描述再分割,提升音频视觉定位的准确与可解释性。

Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation

  • 分三步推理:先理解描述,再粗定位,最后精确分割
  • 在两个基准上达到当前最佳性能,尤其在复杂语义下表现优异
  • 适合需要高可解释性的跨模态定位任务

参照式音视频分割(Ref-AVS)旨在根据参考表达从有声视频中分割出目标对象。以往方法依赖多模态融合学习隐式嵌入,驱动可调的SAM/SAM2解码器进行分割,需强像素级监督且缺乏可解释性。本文提出TGS-Agent,将任务分解为‘思考-定位-分割’流程,模拟人类推理过程:首先通过多模态分析识别被指对象,再进行粗粒度定位与精分割。为此,我们构建了带显式对象感知思维链的指令微调数据集,训练了多模态语言模型Ref-Thinker,用于生成对象描述作为显式提示,输入Grounding-DINO与SAM2,实现无需像素级监督的定位与分割。此外,我们提出了R²-AVSBench新基准,包含语言多样且需推理的参考表达,以更严格评估模型泛化能力。实验表明,该方法在标准Ref-AVSBench与新提出的R²-AVSBench上均取得领先效果。代码将开源于https://github.com/jasongief/TGS-Agent。

原文摘要 · Abstract (English)

Referring Audio-Visual Segmentation (Ref-AVS) aims to segment target objects in audible videos based on given reference expressions. Prior works typically rely on learning latent embeddings via multimodal fusion to prompt a tunable SAM/SAM2 decoder for segmentation, which requires strong pixel-level supervision and lacks interpretability. From a novel perspective of explicit reference understanding, we propose TGS-Agent, which decomposes the task into a Think-Ground-Segment process, mimicking the human reasoning procedure by first identifying the referred object through multimodal analysis, followed by coarse-grained grounding and precise segmentation. To this end, we first propose Ref-Thinker, a multimodal language model capable of reasoning over textual, visual, and auditory cues. We construct an instruction-tuning dataset with explicit object-aware think-answer chains for Ref-Thinker fine-tuning. The object description inferred by Ref-Thinker is used as an explicit prompt for Grounding-DINO and SAM2, which perform grounding and segmentation without relying on pixel-level supervision. Additionally, we introduce R\textsuperscript{2}-AVSBench, a new benchmark with linguistically diverse and reasoning-intensive references for better evaluating model generalization. Our approach achieves state-of-the-art results on both standard Ref-AVSBench and proposed R\textsuperscript{2}-AVSBench. Code will be available at https://github.com/jasongief/TGS-Agent.

音视频分割多模态推理可解释性提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。