用大模型评估音频描述质量，更准且能解释原因。
CLAIR-A: Leveraging Large Language Models to Judge Audio Captions
- 让大模型直接判断音频描述的语义距离，零样本无需训练。
- 在Clotho-Eval上比最优通用指标高11%准确率，比专用指标高5.8%。
- 评分理由更可信，人类评价其解释质量提升30%。
自动音频描述（AAC）任务要求模型为音频输入生成自然语言描述。评估这些生成的描述是一项复杂任务，需综合考虑听觉场景理解、声音对象推断、时间连贯性及环境背景等多重因素。现有方法多聚焦单一维度，难以提供与人类判断一致的综合评分。本文提出CLAI R-A，一种简单灵活的方法，利用大语言模型（LLM）的零样本能力，通过直接询问模型对候选音频描述的语义距离得分来评估其质量。在评估中，CLAIR-A相比传统指标更贴近人类判断，在Clotho-Eval数据集上较领域专用的FENSE指标相对提升5.8%，最高超过最佳通用度量11%。此外，该方法可生成更具透明度的评分理由，人类评价显示其解释质量比基线方法高出最多30%。代码已公开于https://github.com/DavidMChan/clair-a。
原文摘要 · Abstract (English)
The Automated Audio Captioning (AAC) task asks models to generate natural language descriptions of an audio input. Evaluating these machine-generated audio captions is a complex task that requires considering diverse factors, among them, auditory scene understanding, sound-object inference, temporal coherence, and the environmental context of the scene. While current methods focus on specific aspects, they often fail to provide an overall score that aligns well with human judgment. In this work, we propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models (LLMs) to evaluate candidate audio captions by directly asking LLMs for a semantic distance score. In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics, with a 5.8% relative accuracy improvement compared to the domain-specific FENSE metric and up to 11% over the best general-purpose measure on the Clotho-Eval dataset. Moreover, CLAIR-A offers more transparency by allowing the language model to explain the reasoning behind its scores, with these explanations rated up to 30% better by human evaluators than those provided by baseline methods. CLAIR-A is made publicly available at https://github.com/DavidMChan/clair-a.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。