对比了大模型与语音识别的融合方式，发现浅层融合在常见数据上更优。
LLMs and Speech: Integration vs. Combination
- 用浅层融合和紧密集成两种方式结合声学模型与大语言模型
- 在标准数据集上浅层融合始终优于紧密集成，尤其在域内数据表现更好
- 紧密集成在跨领域数据上表现更佳，适合处理多样语音场景
本文研究了利用大语言模型（LLM）进行自动语音识别（ASR）的不同方法，重点比较了将声学模型（AM）与LLM紧密集成（即语音大模型）与传统浅层融合的差异，并对不同标签单元和LLM规模进行了消融实验。对于紧密集成，进一步考察了注意力接口、编码器下采样和长度归一化的影响。此外，通过联合使用CTC模型以缓解语音大模型的幻觉问题并提出有效优化方案。在LibriSpeech和Loquacious上进行训练与评估，并额外在HuggingFace ASR排行榜上测试。结果表明，在不同模型规模下，浅层融合在域内数据上始终优于紧密集成，强调了构建强浅层融合基线的重要性；但在更具异构性的HuggingFace ASR排行榜上，集成前缀的LLM平均字错误率更低，优势集中在域外数据上。
原文摘要 · Abstract (English)
In this work, we study different approaches to utilize large language models (LLMs) for automatic speech recognition (ASR). Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion and provide ablations on the effect of different label units and LLM sizes. For tight integration, we further examine the effect of attention interfaces, encoder downsampling, and length normalization. Furthermore, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations. We train and evaluate on LibriSpeech and Loquacious and additionally evaluate on the HuggingFace ASR leaderboard. Across model sizes, we find that shallow fusion consistently outperforms tight integration of AM and LLM on in-domain data, highlighting the importance of strong shallow-fusion baselines when evaluating speech LLMs for ASR. On the more heterogeneous HuggingFace ASR leaderboard, however, the integrated prefix LLM achieves lower average WER than shallow fusion, with gains concentrated on out-of-domain corpora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。