MACE通过融合音频与字幕信息，更准确评估音频描述质量。
MACE: Leveraging Audio for Evaluating Audio Captioning Systems
- 结合音频信号与字幕内容，加权计算生成质量。
- 在AudioCaps-Eval和Clotho-Eval上分别提升3.28%和4.36%准确率。
- 适合需要真实人类评价一致性的音频描述系统评测。
自动音频描述（AAC）任务旨在用自然语言描述音频信号。传统评估依赖于ROUGE、BLEU等文本生成指标，或SPICE、CIDEr等图像字幕指标，以及Sentence-BERT相似性，但这些方法仅比较生成字幕与人工参考之间的文本相似性，忽略了音频本身的信息。本文提出MACE（多模态音频-字幕评估），一种将音频信号与参考字幕及预测字幕相结合的新型评估指标，并引入流畅性惩罚机制。实验表明，相较于传统指标，MACE在预测人类质量判断方面表现更优：在AudioCaps-Eval和Clotho-Eval数据集上，分别较FENSE指标相对提升3.28%和4.36%。此外，其在音频描述评估任务中显著优于所有已有指标。代码已开源至https://github.com/satvik-dixit/mace。
原文摘要 · Abstract (English)
The Automated Audio Captioning (AAC) task aims to describe an audio signal using natural language. To evaluate machine-generated captions, the metrics should take into account audio events, acoustic scenes, paralinguistics, signal characteristics, and other audio information. Traditional AAC evaluation relies on natural language generation metrics like ROUGE and BLEU, image captioning metrics such as SPICE and CIDEr, or Sentence-BERT embedding similarity. However, these metrics only compare generated captions to human references, overlooking the audio signal itself. In this work, we propose MACE (Multimodal Audio-Caption Evaluation), a novel metric that integrates both audio and reference captions for comprehensive audio caption evaluation. MACE incorporates audio information from audio as well as predicted and reference captions and weights it with a fluency penalty. Our experiments demonstrate MACE's superior performance in predicting human quality judgments compared to traditional metrics. Specifically, MACE achieves a 3.28% and 4.36% relative accuracy improvement over the FENSE metric on the AudioCaps-Eval and Clotho-Eval datasets respectively. Moreover, it significantly outperforms all the previous metrics on the audio captioning evaluation task. The metric is opensourced at https://github.com/satvik-dixit/mace
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。