用统一大模型实现低延迟语音翻译,无需额外分段模型。
StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model
- 用语音思维链引导大模型分阶段生成,一体化完成分段、决策与翻译。
- 在多个数据集上达到当前最佳性能,延迟更低且译文更准确。
- 适合需要实时翻译的场景,如会议同传或直播字幕生成。
流式语音翻译(StreamST)需在持续接收语音输入时确定合适的生成时机(即策略),以平衡低延迟与高翻译质量。现有方法多基于句段级语音切分,称为同步语音翻译(SimulST),实际应用中需依赖分段模型实现流式翻译,而截断的语音片段限制了模型上下文信息,导致策略学习困难。为此,本文提出StreamUni,通过统一的大语音语言模型(LSLM)实现流式语音翻译。具体地,引入语音思维链(CoT)引导模型生成多阶段输出,利用该输出同时完成语音分段、策略决策与翻译生成,无需大规模专用策略训练。此外,设计了一种流式思维链训练方法,仅用少量思维链数据即可提升低延迟下的决策与生成能力。实验表明,该方法在多个流式语音翻译任务上达到领先性能。
原文摘要 · Abstract (English)
Streaming speech translation (StreamST) requires determining appropriate timing, known as policy, to generate translations while continuously receiving source speech inputs, balancing low latency with high translation quality. However, existing StreamST methods typically operate on sentence-level speech segments, referred to as simultaneous speech translation (SimulST). In practice, they require collaboration with segmentation models to accomplish StreamST, where the truncated speech segments constrain SimulST models to make policy decisions and generate translations based on limited contextual information. Moreover, SimulST models struggle to learn effective policies due to the complexity of speech inputs and cross-lingual generation. To address these challenges, we propose StreamUni, which achieves StreamST through a unified Large Speech-Language Model (LSLM). Specifically, StreamUni incorporates speech Chain-of-Thought (CoT) in guiding the LSLM to generate multi-stage outputs. Leveraging these multi-stage outputs, StreamUni simultaneously accomplishes speech segmentation, policy decision, and translation generation, completing StreamST without requiring massive policy-specific training. Additionally, we propose a streaming CoT training method that enhances low-latency policy decisions and generation capabilities using limited CoT data. Experiments demonstrate that our approach achieves state-of-the-art performance on StreamST tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。