让语音大模型实时翻译，兼顾速度与准确率。
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation
- 离线训练语音大模型，推理时用策略动态生成译文。
- 在相同延迟下，语音识别准确率提升3点，翻译更准。
- 适合需要低延迟语音翻译的场景，如会议实时转译。
同步语音翻译（SST）在接收流式语音输入的同时输出翻译结果，需在翻译质量与延迟间取得平衡。尽管大语言模型（LLM）已扩展至处理语音模态，但流式处理仍具挑战性，因语音需作为完整生成过程的前缀提示。本文提出 SimulS2S-LLM，通过离线训练语音大模型，并在测试时采用策略引导同步推理。该方法通过提取具备边界感知能力的语音提示，缓解训练与推理间的不匹配问题，使模型更贴近真实文本输入数据分布。SimulS2S-LLM 实现同步语音到语音翻译（Simul-S2ST），通过预测离散语音标记并使用预训练声码器合成输出语音。设计了增量式束搜索，在不增加延迟的前提下扩大语音标记预测的搜索空间。在 CVSS 语音数据集上的实验表明，该方法在使用相同训练数据的情况下，相比现有方法在同等延迟下提升了 3 点 ASR-BLEU 分数，展现出更优的质量-延迟权衡。
原文摘要 · Abstract (English)
Simultaneous speech translation (SST) outputs translations in parallel with streaming speech input, balancing translation quality and latency. While large language models (LLMs) have been extended to handle the speech modality, streaming remains challenging as speech is prepended as a prompt for the entire generation process. To unlock LLM streaming capability, this paper proposes SimulS2S-LLM, which trains speech LLMs offline and employs a test-time policy to guide simultaneous inference. SimulS2S-LLM alleviates the mismatch between training and inference by extracting boundary-aware speech prompts that allows it to be better matched with text input data. SimulS2S-LLM achieves simultaneous speech-to-speech translation (Simul-S2ST) by predicting discrete output speech tokens and then synthesising output speech using a pre-trained vocoder. An incremental beam search is designed to expand the search space of speech token prediction without increasing latency. Experiments on the CVSS speech data show that SimulS2S-LLM offers a better translation quality-latency trade-off than existing methods that use the same training data, such as improving ASR-BLEU scores by 3 points at similar latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。