首个基于大模型的流式语音转写系统，实时识别说话人与内容。
VibeVoice-ASR-Streaming Technical Report
- 用固定音频块+少量前瞻音频和历史文本，实现边听边输出说话人信息。
- 7B模型在5个数据集上平均词错率最低，13个场景中12个说话人识别最优。
- 适合低延迟语音助手、实时会议记录等需要即时说话人分离的应用。
传统说话人归属语音识别（ASR）系统将语音识别与说话人聚类视为两个独立任务。近年来，如VibeVoice-ASR等端到端模型已将两者统一于单一模型中。然而，现有统一模型仍主要支持离线识别，难以满足实时语音助手与代理的低延迟需求。为此，我们提出VibeVoice-ASR-Streaming，是首个基于大语言模型（LLM）的流式说话人归属语音识别方法。该方法对固定大小的音频块、少量前瞻音频及先前文本进行交错处理，使模型能在语音输入过程中实时生成“谁说了什么”的结果，无需独立的说话人聚类阶段。在转录准确率方面，我们的7B模型在五个评估集上的平均词错误率（WER）和字符错误率（CER）均达到最低；在说话人归属性能上，在13个评估设置中有12个达到最佳或并列最佳。我们已发布1.5B和7B模型权重及推理代码。
原文摘要 · Abstract (English)
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。