arXiv:2505.19937cs.CLcs.SD2025-05被引 3

提出自动评估语音大模型音文对齐质量的无训练评分方法。

ALAS: An Automatic Latent Alignment Score for Audio Language Models

  • 通过对比隐藏层音文表征相似度,量化模型内部对齐程度。
  • 在多个任务上发现对齐深度与模型结构、任务需求相关。
  • 可检测模型高分但未真实理解音频的虚假表现,适合开发者验证。

大型语言模型扩展为语音-语言模型(Speech-LLMs)后,其学习到的音文对齐质量直接影响下游语音理解行为。尽管融合策略不断增多,却缺乏统一衡量语音-文本内部绑定效果的方法。本文提出 ALAS(自动潜在对齐评分),一种无需训练、不依赖特定任务的模型无关指标,通过探测模型各层隐藏状态,计算音文表征间的跨模态余弦相似度,并以 Whisper 提取的参考作为基准。ALAS 仅需一次前向传播和现成的 ASR 参考,且校准至可解释的均匀基线,支持跨任务比较。在四个开源语音大模型(AF3、Qwen2-Audio、Qwen-Omni、SALMONN)上应用于情感识别(IEMOCAP)、开放式问答(LibriSQA)和多选音频理解(MMAU-speech)任务,结果表明对齐深度与强度反映模型音编码器设计及任务中声学与语义的需求差异,且 ALAS 能追踪但不重复任务准确率,揭示出部分模型虽得分高却未真正基于音频的理解。ALAS 已开源,供研究者用于自检或新任务测试。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are extended into Speech-LLMs, and the quality of the audio--text alignment they learn affects most downstream Spoken Language Understanding (SLU) behavior. Yet despite a growth of fusion strategies, there is no standard way to measure how well a Speech-LLM internally binds audio frames to text tokens. We introduce ALAS (Automatic Latent Alignment Score), a model and task-agnostic metric that probes the LLM's per-layer hidden states, scoring the cross-modal cosine similarity between audio and text representations against a Whisper-derived reference. ALAS needs only a frozen forward pass and an off-the-shelf ASR reference, with no training or fitted classifier, and is calibrated to an interpretable uniform baseline comparable across tasks. Applying ALAS to four open-source Speech-LLMs (AF3, Qwen2-Audio, Qwen-Omni, SALMONN) across emotion recognition (IEMOCAP), open-ended SQA (LibriSQA), and multi-choice audio understanding (MMAU-speech), we find that the depth and strength of alignment reflect each model's audio-encoder design and the acoustic-versus-semantic demands of the task, and that ALAS tracks but does not duplicate task accuracy, exposing models that score well without genuinely grounding in the audio. We release ALAS as an open-source library so that practitioners can probe their own Speech-LLMs or try it on new tasks.

语音大模型对齐评估无训练方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。