用大模型零训练实现音频引导的视频物体分割
MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge
- 不训练模型，用多模态大模型理解音频与视觉关系
- 结合SAM生成精确物体掩码，性能达比赛第三
- 适合想快速部署音频分割系统的开发者
本文提出一种无需训练的音频引导视频物体分割框架，将多模态大语言模型（MLLM）与基于SAM的分割模型结合。将任务分解为多个阶段，并为每阶段匹配合适的基础模型。无需额外训练或特定任务微调，利用MLLM强大的多模态推理能力建模文本-视觉对应关系，同时使用SAM-based模型生成精准物体掩码。该框架在第八届LSVOS挑战赛的MeViS-Audio赛道中取得优异表现，验证了基础模型在音频引导视频分割中的有效性。
原文摘要 · Abstract (English)
In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models. We decompose the task into several stages and identify suitable foundation models for each stage. Without introducing additional model training or task-specific fine-tuning, our approach leverages the strong multimodal reasoning capabilities of MLLMs to model text-visual correspondence and employs SAM-based models for accurate object mask generation. The proposed framework demonstrates the effectiveness of leveraging foundation models for audio-guided video segmentation and achieves competitive performance in the MeViS-Audio Track of the 8th LSVOS Challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。