用语音转文字+现成模型，高效实现音频驱动的视频物体分割
2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA
- 先将音频转为动作描述文本，再用预训练文本分割模型生成像素级结果
- 在PVUW挑战中取得80.7分，排名第二，且计算开销更低
- 适合资源有限但需处理模糊音频的视频理解场景
基于音频的视频物体分割旨在根据音频线索定位并分割视频中的目标对象，需要精确理解外观与运动。现有方法通过融合音视频特征扩展多模态大模型（MLLM）实现端到端定位，但存在计算成本高、时序对齐困难、依赖大规模音视频配对数据等问题。为此，我们提出ASR-SaSaSa2VA，一种资源高效的音频引导视频分割框架。核心思路是利用自动语音识别（ASR）模型将音频输入转换为文本形式的动作描述，再借助预训练的文本导向视频指代分割模型（如SaSaSa2VA）进行像素级预测。为进一步提升鲁棒性，引入一个目标不存在检测模块，由微调后的音频型多模态大模型实现，用于过滤不指代任何目标的音频片段。该设计使系统能有效利用强预训练模型，同时应对模糊或无关音频。本方法在第五届PVUW挑战赛（MeViS-v2-Audio赛道）中获得80.7分，位列第二。
原文摘要 · Abstract (English)
Audio-based video object segmentation aims to locate and segment objects in videos conditioned on audio cues, requiring precise understanding of both appearance and motion. Recent audio-driven video segmentation methods extend MLLMs by fusing audio and visual features for end-to-end localization. Despite their promise, these approaches are computationally intensive, struggle with aligning temporal audio cues to dynamic video content, and depend on large paired audio-video datasets. To address these challenges, we present ASR-SaSaSa2VA, a resource-efficient framework for audio-guided video segmentation. The key idea is to convert audio inputs into textual motion descriptions via automatic speech recognition (ASR) models and then leverage pre-trained text-based referring video segmentation models (e.g., SaSaSa2VA) for pixel-level predictions. To further enhance robustness, we incorporate a no-target expression detection module, implemented by a fine-tuned audio-based MLLM, which filters out audio clips that do not refer to any target object. This design allows the system to exploit strong pre-trained models while effectively handling ambiguous or irrelevant audio inputs. Our approach achieves a final score of 80.7 in the 5th PVUW Challenge (MeViS-v2-Audio track), earning the second-place ranking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。