用文本桥接音频视觉,实现精准分割
TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models
- 用文本作为桥梁,融合多模态大模型与分割模型
- 在单源、多源及零样本场景下表现优异
- 适合需要跨模态对齐的多媒体分割任务
音频-视觉分割(AVS)面临有效对齐音视频模态的根本挑战。尽管近期方法利用基础模型缓解数据稀缺问题,但通常依赖单模态知识或以现成方式结合基础模型,未能解决跨模态对齐难题。本文提出TAViS,一种新框架,将多模态基础模型(ImageBind)的跨模态对齐能力与分割基础模型(SAM2)的精细分割能力相结合。然而,有效整合两者存在两大挑战:因特征空间差异导致的知识迁移困难,以及仅使用分割损失监督的不足。为此,我们引入文本桥接设计,包含两个关键组件:(1) 文本桥接混合提示机制,其中伪文本提供类别原型信息,同时保留音视频输入的模态特异性;(2) 利用文本作为桥梁,对齐音视频模态中的共享语义概念的对齐监督策略。该方法在单源、多源及语义数据集上均取得优异性能,并在零样本设置下表现突出。
原文摘要 · Abstract (English)
Audio-Visual Segmentation (AVS) faces a fundamental challenge of effectively aligning audio and visual modalities. While recent approaches leverage foundation models to address data scarcity, they often rely on single-modality knowledge or combine foundation models in an off-the-shelf manner, failing to address the cross-modal alignment challenge. In this paper, we present TAViS, a novel framework that \textbf{couples} the knowledge of multimodal foundation models (ImageBind) for cross-modal alignment and a segmentation foundation model (SAM2) for precise segmentation. However, effectively combining these models poses two key challenges: the difficulty in transferring the knowledge between SAM2 and ImageBind due to their different feature spaces, and the insufficiency of using only segmentation loss for supervision. To address these challenges, we introduce a text-bridged design with two key components: (1) a text-bridged hybrid prompting mechanism where pseudo text provides class prototype information while retaining modality-specific details from both audio and visual inputs, and (2) an alignment supervision strategy that leverages text as a bridge to align shared semantic concepts within audio-visual modalities. Our approach achieves superior performance on single-source, multi-source, semantic datasets, and excels in zero-shot settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。