实时语音识别模型在480毫秒延迟下达到离线水平精度。
Voxtral Realtime
- 端到端训练的流式模型，显式对齐音视频与文本流。
- 480毫秒延迟下性能媲美广泛使用的Whisper模型。
- 支持13种语言，开源可商用，适合实时语音处理场景。
我们提出Voxtral Realtime，一种原生流式自动语音识别模型，在亚秒级延迟下实现与离线转录相当的性能。与通过分块或滑动窗口改造离线模型的方法不同，Voxtral Realtime从头开始端到端训练以支持流式输入，并显式对齐音频与文本流。其架构基于延迟流建模框架，引入新的因果音频编码器和自适应RMS归一化（Ada RMS-Norm）以增强延迟条件建模能力。我们在覆盖13种语言的大规模数据集上进行预训练。在480毫秒延迟下，Voxtral Realtime的性能与目前应用最广泛的离线系统Whisper相当。模型权重已按Apache 2.0许可证发布。
原文摘要 · Abstract (English)
We introduce Voxtral Realtime, a natively streaming automatic speech recognition model that matches offline transcription quality at sub-second latency. Unlike approaches that adapt offline models through chunking or sliding windows, Voxtral Realtime is trained end-to-end for streaming, with explicit alignment between audio and text streams. Our architecture builds on the Delayed Streams Modeling framework, introducing a new causal audio encoder and Ada RMS-Norm for improved delay conditioning. We scale pretraining to a large-scale dataset spanning 13 languages. At a delay of 480ms, Voxtral Realtime achieves performance on par with Whisper, the most widely deployed offline transcription system. We release the model weights under the Apache 2.0 license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。