用少样本数据训练端到端多说话人语音识别模型，兼顾识别与说话人区分精度。
Balancing ASR and diarization in end-to-end LLMs for multi-talker speech recognition
- 双编码器分离语义与说话人特征，特征交叉输入大模型
- 在AliMeeting上提升18%，Aishell4上提升24%的识别准确率
- 适合资源有限但需高精度多说话人场景的语音系统开发者
多说话人语音识别通常采用先识别后分说话人的流水线系统。近年来，基于大语言模型（LLM）的方法通过联合建模语义与说话人信息展现出潜力，但往往需要大规模标注的多说话人数据，成本高昂。本文研究如何在仅有少量真实录音数据的情况下高效训练LLM系统，同时保持高说话人识别精度。提出四种策略：(1) 双编码器架构分别提取语义与说话人特征；(2) 特征交错格式将两类特征合并为模型输入；(3) 长度感知说话人身份损失以增强分说话人能力；(4) 自适应阈值策略计算ASR损失，缓解语音重叠导致的幻觉。这些方法有效平衡了语音识别与说话人区分的训练。实验显示，该系统优于开源基线，在AliMeeting数据集上相对提升18%，在Aishell4数据集上提升24%。
原文摘要 · Abstract (English)
Multi-talker speech recognition is often addressed by combining automatic speech recognition (ASR) and speaker diarization in a pipeline system. Recently, LLM-based approaches have shown promise by jointly modeling semantic and speaker information, but they typically require large-scale multi-talker corpora that are costly to annotate. In this paper, we investigate how to efficiently train an LLM-based system with limited real-recorded data while maintaining high accuracy in speaker attribution. We propose several strategies: (1) a dual-encoder architecture to extract semantic and speaker features, (2) a feature interleaving format to merge these features as the inputs to the LLM, (3) a length-aware speaker ID loss to enhance diarization capability, and (4) an adaptive threshold strategy for ASR loss computation to mitigate hallucinations caused by speech overlaps. These strategies balance training between ASR and diarization tasks. Our system outperforms open-source baseline approaches, achieving relative improvements of 18% on the AliMeeting corpus and 24% on the Aishell4 corpus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。