arXiv:2602.03892cs.CVcs.AI2026-02被引 2

无需真值标注,用多模态模型自动评估音视频分割掩码质量。

Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation

  • 基于多模态大模型,结合音视频与掩码信息进行推理判断。
  • 能准确预测掩码与真值的重合度并识别几何/语义错误类型。
  • 适合需要自动化质量检测的音视频分割系统开发者使用。

语言描述的音视频分割(Ref-AVS)旨在通过联合推理视频、音频和文本,定位自然语言描述的目标对象。除了生成分割掩码外,提供丰富且可解释的掩码质量诊断仍鲜有研究。本文提出在 Ref-AVS 场景下的掩码质量评估任务(MQA-RefAVS),该任务不依赖推理时的真值标注,仅根据音视频语言输入与候选掩码,估计其与未观测真值的交并比(IoU),识别错误类型,并推荐可操作的质量控制决策。为此,我们构建了涵盖多种典型错误模式的基准数据集 MQ-RAVSBench。进一步提出 MQ-Auditor,一个基于多模态大语言模型(MLLM)的审计器,显式融合多模态线索与掩码信息,实现定量与定性双重评估。大量实验表明,MQ-Auditor优于多个开源及商用 MLLM,且可集成至现有 Ref-AVS 系统中,有效检测分割失败并支持下游改进。代码与数据将公开于 https://github.com/jasongief/MQA-RefAVS。

原文摘要 · Abstract (English)

Language-referred audio-visual segmentation (Ref-AVS) aims to segment target objects described by natural language by jointly reasoning over video, audio, and text. Beyond generating segmentation masks, providing rich and interpretable diagnoses of mask quality remains largely underexplored. In this work, we introduce Mask Quality Assessment in the Ref-AVS context (MQA-RefAVS), a new task that evaluates the quality of candidate segmentation masks without relying on ground-truth annotations as references at inference time. Given audio-visual-language inputs and each provided segmentation mask, the task requires estimating its IoU with the unobserved ground truth, identifying the corresponding error type, and recommending an actionable quality-control decision. To support this task, we construct MQ-RAVSBench, a benchmark featuring diverse and representative mask error modes that span both geometric and semantic issues. We further propose MQ-Auditor, a multimodal large language model (MLLM)-based auditor that explicitly reasons over multimodal cues and mask information to produce quantitative and qualitative mask quality assessments. Extensive experiments demonstrate that MQ-Auditor outperforms strong open-source and commercial MLLMs and can be integrated with existing Ref-AVS systems to detect segmentation failures and support downstream segmentation improvement. Data and codes will be released at https://github.com/jasongief/MQA-RefAVS.

音视频分割多模态质量评估大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。