用对话模型实现无边界语音的实时翻译，降低延迟且保持质量
InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model
- 将同步翻译建模为多轮对话，无需预切分语音流
- 在MuST-C数据上训练，延迟降低0.5~1秒，质量不变
- 自研缓存管理策略，适合实际部署的长语音场景
无边界流式语音的同步翻译仍具挑战，需有效处理历史语音上下文和已译内容，以平衡翻译质量与延迟（含计算开销）。多数已有方法依赖预分割语音，限制了实际应用。本文提出InfiniSST，将同步翻译建模为多轮对话任务，实现无边界语音的无缝翻译。训练时基于MuST-C数据集，通过多延迟增强构建翻译轨迹与鲁棒段；并设计键值（KV）缓存管理策略，提升推理效率。在MuST-C En-Es、En-De 和 En-Zh 数据集上的实验表明，InfiniSST 在保持与基线相当翻译质量的前提下，计算感知延迟降低0.5至1秒。消融实验验证了数据构造与缓存管理策略的有效性。代码与演示已开源：https://github.com/LeiLiLab/InfiniSST
原文摘要 · Abstract (English)
Simultaneous translation of unbounded streaming speech remains a challenging problem due to the need for effectively processing the history speech context and past translations so that quality and latency, including computation overhead, can be balanced. Most prior works assume pre-segmented speech, limiting their real-world applicability. In this paper, we propose InfiniSST, a novel approach that formulates SST as a multi-turn dialogue task, enabling seamless translation of unbounded speech. We construct translation trajectories and robust segments from MuST-C with multi-latency augmentation during training and develop a key-value (KV) cache management strategy to facilitate efficient inference. Experiments on MuST-C En-Es, En-De, and En-Zh demonstrate that InfiniSST reduces computation-aware latency by 0.5 to 1 second while maintaining the same translation quality compared to baselines. Ablation studies further validate the contributions of our data construction and cache management strategy. We release the code and demo at https://github.com/LeiLiLab/InfiniSST
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。