构建跨时长与认知深度的音频理解评测基准
AudioSpan: Spanning the Duration and Depth of Audio Comprehension
- 设计跨10分钟至2小时音频的多层级问答体系
- 3240道题覆盖感知、理解、推理三层次,揭示模型瓶颈在信息提取
- 自动化流水线生成带对抗反馈的高质量问题,适合评估长音频模型
通用音频理解涵盖从秒级到小时级的语音、音效和音乐,由日益多模态的大规模音频-语言模型(LALMs)推动。然而现有评测仍依赖秒级片段,导致得分饱和、模型性能趋同;近期长时研究虽延长了时长,但评估方式仍沿用短片段模式。本文提出AudioSpan,一个同时跨越时长与认知深度的评测基准:将10分钟至2小时以上的音频与3240个问题配对,覆盖感知、理解、推理三个认知层级。通过两种路径生成问题——原生问答(Native QA)从音频内容中提取问题,采用多选与开放题结合、详细评分标准;锚点问答(Anchor QA)注入真实信息,在音频中植入声学锚点,构建从感知到推理的链式任务,仅在首次错误处评分。整个流程由结构化描述、问答生成与对抗性批评反馈构成的全自动管道实现。对12个LALMs的评估发现,困难主要出现在推理前:从冗长、重复的信号中提炼少数相关事实。该难度随音频长度增加而加剧,尤其在时间定位任务上表现最弱。AudioSpan已开源于https://huggingface.co/datasets/holvan/AudioSpan。
原文摘要 · Abstract (English)
General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce AudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on AudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. AudioSpan is available at https://huggingface.co/datasets/holvan/AudioSpan.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。