将大模型语义先验融入编码器，实现快速多说话人语音识别
Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing
- 用大模型做教师指导，将语义先验蒸馏到编码器中
- 三说话人场景下性能显著提升，实时因子（RTF）大幅降低
- 支持可变说话人数，动态路由选择解码分支
大型语言模型（LLM）具备强大的语义先验，可提升多说话人自动语音识别（MT-ASR）性能，但将其作为自回归解码器计算开销大且在严重重叠下表现脆弱。本文提出一种编码器仅结构的MT-ASR框架，将LLM适配为多说话人条件，并在训练中将其语义指导蒸馏至编码器，推理时保留快速的CTC式解码。模型采用后编码器分离器与序列化CTC生成有序转录，利用改进的基于LLM的SOT目标作为多说话人感知的教师信号，显式正则化混合语音表示。为进一步支持可变说话人数，引入说话人数量头（Talker-Count Head），预测说话人数并动态选择解码分支。在LibriMix数据集上的实验表明，该模型在双说话人条件下达到与基于LLM系统相当的性能，而在三说话人条件下有显著提升，且实时因子（RTF）明显减小。
原文摘要 · Abstract (English)
Large language models (LLMs) provide strong semantic priors that can improve multi-talker automatic speech recognition (MT-ASR), but using an LLM as an autoregressive decoder is computationally expensive and remains fragile under heavy overlap. In this paper, we propose an encoder-only MT-ASR framework that adapts an LLM to multi-talker conditioning and distills its semantic guidance into the encoder during training, while retaining fast CTC-style decoding at inference. Our model employs a post-encoder separator with serialized CTC to produce talker-ordered transcripts, and leverages an adapted LLM-based SOT objective as a multi-talker-aware teacher signal to explicitly regularize mixed-speech representations. To further support variable numbers of talkers, we introduce a Talker-Count Head that predicts the talker count and dynamically selects the appropriate decoding branch. Experiments on LibriMix show that the proposed encoder-only model achieves comparable performance to LLM-based systems in the two-talker condition, while delivering significant improvements in the three-talker condition with significant small RTF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。