评估语音大模型在已知任务中的生成调用价值，发现非生成方法已可达到顶尖性能。
Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluation
- 将语音模型评估设计为可控的调用决策问题，区分声学证据与生成调用的作用
- 无生成调用的监督模型（如CLAP、WavLM）准确率达0.850以上，优于含生成调用的策略
- 生成调用仅带来微小增益（0.004），适合追求高效推理的研究者参考
语音和音频大模型常通过波形提示是否优于自动语音识别（ASR）转录文本进行评估。对于已知封闭集任务，这种比较混淆了声学证据获取与调用生成音频模型的双重因素。本文将其作为受控调用决策问题进行评估。每个样本中，策略在保留转录标签、使用对比语言-音频预训练（CLAP）、音频频谱变换器（AST）、WavLM的编码器特征，或调用Qwen2-Audio、Qwen2.5-Omni、MOSS-Audio之间选择。关键消融移除所有生成调用，保持选择器和开发协议不变。在VocalSound数据集上，转录文本准确率为0.296，需波形信息。但监督的CLAP和WavLM控制组分别达到0.850和0.854。包含生成调用的选择器达0.925准确率，使用12.5%的生成调用；而对应无生成调用的选择器为0.921（配对差异0.004；95%置信区间[-0.025,0.033]）。共识与堆叠特征提升弱选择器表现，但无法超越最强无生成控制组。对于已知任务的终点宣称，关键在于在已有转录与编码器证据后，生成调用的边际价值。
原文摘要 · Abstract (English)
Speech and audio LLMs are often evaluated by asking whether a waveform prompt beats an automatic speech recognition (ASR) transcript. For known closed-set tasks, that comparison conflates two factors: access to acoustic evidence and the need to call a generative audio model. We evaluate this distinction as a controlled call-decision problem. For each example, a policy chooses among keeping a transcript label, using encoder evidence from Contrastive Language-Audio Pretraining (CLAP), Audio Spectrogram Transformer (AST), or WavLM, and calling Qwen2-Audio, Qwen2.5-Omni, or MOSS-Audio; the decisive ablation removes all generative actions while keeping the selector and development protocol fixed. On VocalSound, transcripts reach 0.296 accuracy, so waveform information is needed. Yet supervised CLAP and WavLM controls reach 0.850 and 0.854 with no generative audio calls. A selector with generative actions reaches 0.925 accuracy using 12.5% calls, compared with 0.921 for the matched no-call selector (paired difference 0.004; 95% CI [-0.025,0.033]). Agreement and stacking features improve weaker selectors but do not beat the strongest no-call control. For known-task endpoint claims, the relevant quantity is the marginal value of the generative call after transcript and encoder evidence have already been used.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。