统一语音大模型端到端实现多语言对话的说话人分离与识别
A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations
- 将说话人分离与语音识别联合建模，采用端到端训练与推理优化
- 在无分段信息条件下，相对基线提升54.87%的tcpWER/tcpCER
- 适用于多语言对话场景，适合需要高集成度语音处理的系统
语音大语言模型（Speech LLM）近年来成为重要范式，拓展了传统大模型在自动语音识别（ASR）和口语对话建模中的能力。然而，其在真实多语言对话中的表现受限于缺乏自然对话数据。为此，MLC-SLM挑战赛提供了多语言对话数据集，并评估模型在两个任务上的表现：带理想分段的ASR（任务I）和无理想分段的联合说话人分离与识别（任务II）。本文聚焦任务II，提出一种统一的语音大模型，实现端到端的说话人分离与语音识别。通过重构训练数据格式并调整推理流程，模型缓解了预分段音频带来的歧义问题，在相同条件下相较基线模型取得54.87%的相对性能提升（tcpWER/tcpCER），排名第八，且使用更小的模型主干。
原文摘要 · Abstract (English)
Speech Large Language Models (Speech LLMs) have emerged as a crucial paradigm in recent years, extending the capabilities of traditional LLMs to speech tasks such as automatic speech recognition (ASR) and spoken dialogue modeling. However, their effectiveness in real-world multilingual conversations remains limited by the scarcity of data that captures natural conversational phenomena. To address this, the MLC-SLM Challenge provides a multilingual conversational dataset and evaluates models on two tasks: ASR with oracle segmentation (Task I) and joint diarization and recognition without oracle information (Task II). In this paper, we focus on Task II and propose a unified speech LLM that jointly performs diarization and ASR in an end-to-end manner. By reformulating the training data format and modifying the inference procedure, our model addresses the ambiguity inherent in pre-segmented audio and achieves a 54.87\% relative improvement in tcpWER/tcpCER over the baseline, ranking 8th overall, despite using a smaller LLM backbone. We also report results from Task I using a fine-tuned speech LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。