用说话人分割结果引导大模型,实现更准的多人对话语音识别
DM-ASR: Diarization-aware Multi-speaker ASR with Large Language Models

- 将多说话人识别转为带说话人和时间条件的对话生成任务
- 在中英文数据集上用小模型就达到顶尖性能,优于联合建模方法
- 适合需要高精度说话人分离的会议记录、访谈转录场景
多说话人自动语音识别旨在转录多人对话,需同时捕捉说什么、谁说的以及何时说的。近期语音-大模型方法展现了统一建模潜力,但联合学习说话人归属、时间结构与词义识别仍困难且需大量数据。当前阶段,利用可靠的说话人分割作为显式结构先验是简化该任务的有效途径。为此,我们提出DM-ASR,一种基于说话人分割的多说话人语音识别框架,将任务重新构造成多轮对话生成过程。给定音频片段和分割结果,DM-ASR将其分解为一系列由说话人和时间条件驱动的查询,每条对应一个说话人在一个时间段内的发言。该形式将说话人-时间结构与语言内容显式解耦,使说话人分割信号与大语言模型的推理能力有效融合。我们还引入可选的词级时间戳预测机制,交错插入词与时间戳标记,生成更丰富的结构化输出,提升转录质量。分析表明,分割系统提供更可靠的说话人身份和段落边界,而大模型擅长语言内容与长程依赖建模,二者互补。在中文和英文基准测试上,该方法以较小模型和训练数据实现强性能,优于或媲美现有统一方法。
原文摘要 · Abstract (English)
Multi-speaker automatic speech recognition (ASR) aims to transcribe conversational speech involving multiple speakers, requiring the model to capture not only what was said, but also who said it and sometimes when it was spoken. Recent Speech-LLM approaches have shown the potential of unified modeling for this task, but jointly learning speaker attribution, temporal structure, and lexical recognition remains difficult and data-intensive. At the current stage, leveraging reliable speaker diarization as an explicit structural prior provides a practical and efficient way to simplify this task. To effectively exploit such priors, we propose DM-ASR, a diarization-aware multi-speaker ASR framework that reformulates the task as a multi-turn dialogue generation process. Given an audio chunk and diarization results, DM-ASR decomposes transcription into a sequence of speaker- and time-conditioned queries, each corresponding to one speaker in one time segment. This formulation converts multi-speaker recognition into a series of structured sub-tasks, explicitly decoupling speaker-temporal structure from linguistic content and enabling effective integration of diarization cues with the reasoning capability of large language models. We further introduce an optional word-level timestamp prediction mechanism that interleaves word and timestamp tokens, yielding richer structured outputs and better transcription quality. Our analysis shows that diarization systems provide more reliable speaker identities and segment-level boundaries, while LLMs excel at modeling linguistic content and long-range dependencies, demonstrating their complementary strengths. Experiments on Mandarin and English benchmarks show that the proposed approach achieves strong performance with relatively small models and training data, while remaining competitive with or outperforming existing unified approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。