构建细粒度音频理解新基准，提升模型对声音细节的感知能力
MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks
- 用多专家模型与思维链推理生成多视角细粒度音频描述
- 引入新评估指标DATE，惩罚泛化表述，奖励具体细节描述
- 适用于评测语音理解模型在复杂场景下的精细感知能力
尽管大型音频-语言模型在开放域音频理解方面取得了进展，但仍难以达到人类级别的细致理解。这一差距主要源于现有基准数据集在标注和评估指标上的局限性，无法可靠区分模型输出的通用性与高度细节性。为此，本文提出MECAT，一个基于多专家构建的细粒度音频理解基准。该基准通过整合专业专家模型分析与思维链大语言模型推理的流水线生成，提供多视角、细粒度的音频描述及开放集问答对。同时，提出新型评估指标DATE（Discriminative-Enhanced Audio Text Evaluation），通过结合单样本语义相似度与跨样本可区分性，惩罚通用词汇并奖励详细描述。对当前先进音频模型的全面评估揭示了其能力和局限。数据与代码已开源：https://github.com/xiaomi-research/mecat
原文摘要 · Abstract (English)
While large audio-language models have advanced open-ended audio understanding, they still fall short of nuanced human-level comprehension. This gap persists largely because current benchmarks, limited by data annotations and evaluation metrics, fail to reliably distinguish between generic and highly detailed model outputs. To this end, this work introduces MECAT, a Multi-Expert Constructed Benchmark for Fine-Grained Audio Understanding Tasks. Generated via a pipeline that integrates analysis from specialized expert models with Chain-of-Thought large language model reasoning, MECAT provides multi-perspective, fine-grained captions and open-set question-answering pairs. The benchmark is complemented by a novel metric: DATE (Discriminative-Enhanced Audio Text Evaluation). This metric penalizes generic terms and rewards detailed descriptions by combining single-sample semantic similarity with cross-sample discriminability. A comprehensive evaluation of state-of-the-art audio models is also presented, providing new insights into their current capabilities and limitations. The data and code are available at https://github.com/xiaomi-research/mecat
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。