用大模型实现低延迟实时语音转文字翻译，仅1-2秒延迟。
Streaming Speech-to-Text Translation with a SpeechLLM
- 大模型同时学生成文本和判断何时足够听
- 多语言测试中翻译质量接近非流式基准
- 适合实时会议、字幕等低延迟场景
传统语音转文字翻译系统通常由语音识别与文本翻译模块组成。将两者整合为语音大模型（SpeechLLM）可利用语音中的副语言信息并减少级联错误。但现有SpeechLLM系统运行缓慢，无法真正流式处理：要么等待完整音频才输出，要么按固定间隔输出，不适用于实际应用。本文提出一种基于大模型的实时流式语音转文字翻译架构，使模型不仅学会生成输出词元，还学会判断是否已获取足够音频信息。系统通过输入语音与输出文本的自动对齐进行训练。在多个语言对上的实验表明，该系统翻译质量接近非流式基线，但延迟仅为1-2秒。
原文摘要 · Abstract (English)
Normally, a system that translates speech into text consists of separate modules for speech recognition and text-to-text translation. Combining those tasks into a SpeechLLM promises to exploit paralinguistic information in the speech and to reduce cascaded errors. But existing SpeechLLM systems are slow since they do not work in a real streaming fashion: they wait for a complete utterance of audio before outputting a translation, or output tokens at fixed intervals, which is not suitable for real applications. This work proposes an LLM-based architecture for real streaming speech-to-text translation. The LLM learns not just to emit output tokens, but also to decide whether it has seen enough audio to do so. The system is trained using automatic alignments of the input speech and the output text. In experiments on different language pairs, the system achieves a translation quality close to the non-streaming baseline, but with a latency of only 1-2 seconds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。