针对语音描述的视频目标分割,提出四阶段处理流程提升准确性。
APRVOS: 1st Place Winner of 5th PVUW MeViS-Audio Track

- 先转写语音为文本,再验证目标是否真实存在
- 生成粗略分割轨迹并动态修正边界与时间一致性
- 适合处理含噪声语音输入的视觉理解任务
本文针对语音表达的视频目标分割(Ref-VOS)任务,提出一种适配MEVIS_Audio场景的系统。该场景中参考表达以语音形式给出,而非清晰文本。相比标准的Sa2VA基线方法,本系统增加两个前端模块:语音转写与视觉存在性验证。首先使用VibeVoice-ASR将长语音输入转化为结构化文本;由于音频查询常含噪声且可能指向视频中不存在的对象,进而引入基于Omni的判断模块,确认目标是否在视觉内容中存在。若判定不存在,则直接输出全零掩码终止流程。否则,将转写文本转换为分割导向提示,输入Sa2VA生成全视频范围的粗略掩码轨迹。该轨迹作为初始语义假设,非最终结果。随后,代理式精修层评估查询可靠性、时间相关性、锚点质量及潜在错误源,必要时调用SAM3以提升空间边界精度和时间一致性。整个框架明确分解任务为:音频转文本、视觉存在性判断、粗分割、代理引导精修,显著优于直接输入噪声语音识别结果给分割模型。
原文摘要 · Abstract (English)
This report presents an Audio-aware Referring Video Object Segmentation (Ref-VOS) pipeline tailored to the MEVIS\_Audio setting, where the referring expression is provided in spoken form rather than as clean text. Compared with a standard Sa2VA-based Ref-VOS pipeline, the proposed system introduces two additional front-end stages: speech transcription and visual existence verification. Specifically, we first employ VibeVoice-ASR to convert long-form spoken input into a structured textual transcript. Since audio-derived queries are inherently noisy and may describe entities that are not visually present in the video, we then introduce an Omni-based judgment module to determine whether the transcribed target can be grounded in the visual content. If the target is judged to be absent, the pipeline terminates early and outputs all-zero masks. Otherwise, the transcript is transformed into a segmentation-oriented prompt and fed into Sa2VA to obtain a coarse mask trajectory over the full video. Importantly, this trajectory is treated as an initial semantic hypothesis rather than a final prediction. On top of it, an agentic refinement layer evaluates query reliability, temporal relevance, anchor quality, and potential error sources, and may invoke SAM3 to improve spatial boundary precision and temporal consistency. The resulting framework explicitly decomposes the MEVIS\_Audio task into audio-to-text conversion, visual existence verification, coarse video segmentation, and agent-guided refinement. Such a staged design is substantially more appropriate for audio-conditioned Ref-VOS than directly sending noisy ASR outputs into a segmentation model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。