用短音频训练的语音大模型，实现长音频实时联合识别与说话人分离。
Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio
- 通过动态更新的说话人提示缓存，实现流式推理时的零样本长音频处理。
- 在20秒内音频上超越强基线，在长音频上优于非流式级联模型。
- 适合会议转录等需实时处理且已有说话人信息的场景。
联合自动语音识别（ASR）与说话人分离旨在回答多说话人场景中“谁说了什么”。本文提出一种端到端语音大语言模型（Speech-LLM）——JEDIS-LLM，仅在20秒以下短音频上训练，却可在无需额外训练的情况下实现长音频的流式推理。该方法基于大语言模型的自回归特性，引入可在线更新的说话人提示缓存（SPC），支持预注册说话人信息的无缝使用。训练中加入词级说话人监督以增强分离能力。实验表明，本系统在20秒内音频上超越Sortformer和Meta-Cat，在长音频上优于非流式级联模型DiarizationLM，且为首个实现零样本流式联合处理的语音大模型，达到当前最优性能。
原文摘要 · Abstract (English)
Joint automatic speech recognition (ASR) and speaker diarization aim to answer the question "who spoke what" in multi-speaker scenarios. In this paper, we present an end-to-end speech large language model (Speech-LLM) for Joint strEamable DIarization and aSr (JEDIS-LLM). The model is trained only on short audio under 20s but is capable of streamable inference on long-form audio without additional training. This is achieved by introducing a Speaker Prompt Cache (SPC) with an on-the-fly update mechanism during chunk-wise streaming inference, inspired by the autoregressive nature of LLMs. The SPC also allows the seamless use of pre-enrolled speaker profiles which is common in many scenarios like meeting transcription. To further enhance diarization capability, we incorporate word-level speaker supervision into the speech encoder during training. Experimental results demonstrate that our system outperforms strong baselines, including Sortformer and Meta-Cat in the local setting on audio up to 20s, and DiarizationLM on long-form audio, despite being fully end-to-end and streamable while DiarizationLM follows a cascaded offline pipeline. To the best of our knowledge, this is the first work enabling zero-shot streamable joint ASR and diarization on long audio using a Speech-LLM trained only on short audio, achieving state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。