无需对齐数据即可实现端到端实时语音翻译，支持多语言快速扩展。
Simultaneous Speech-to-Speech Translation Without Aligned Data
- 通过强化学习优化延迟，跳过传统对齐步骤，训练更简单。
- 在五项跨语言翻译任务中达到顶尖准确率与自然度，延迟更低。
- 仅需不足1000小时语音即可适配新语言，适合多语种应用开发。
同步语音翻译需在实时处理中应对非单调词序依赖。传统方法依赖字级对齐数据进行监督训练，但此类数据难以大规模获取，常依赖语言特异性启发式生成合成对齐，效果不佳。本文提出Hibiki-Zero，完全消除对字级对齐的需求，从根本上简化训练流程，可无缝扩展至语法结构差异大的多种语言，避免设计语言特异性对齐规则的瓶颈。先在句级对齐数据上训练高延迟的语音翻译模型，再采用新颖的GRPO强化学习策略，在降低延迟的同时保持翻译质量。Hibiki-Zero在五项X-to-English任务中实现翻译准确率、延迟、语音传递和自然度的领先表现。此外，我们证明该模型可在少于1000小时语音数据下适配新输入语言。我们提供示例、模型权重、推理代码，并发布包含45小时多语言数据的基准评测集。
原文摘要 · Abstract (English)
Simultaneous speech translation requires translating source speech into a target language in real-time while handling non-monotonic word dependencies. Traditional approaches rely on supervised training with word-level aligned data, which is difficult to collect at scale and thus depends on synthetic alignments using language-specific heuristics that are suboptimal. We propose Hibiki-Zero, which eliminates the need for word-level alignments entirely. This fundamentally simplifies the training pipeline and enables seamless scaling to diverse languages with varying grammatical structures, removing the bottleneck of designing language-specific alignment heuristics. We first train on sentence-level aligned data to learn speech translation at high latency, then apply a novel reinforcement learning strategy using GRPO to optimize latency while preserving translation quality. Hibiki-Zero achieves state-of-the-art performance in translation accuracy, latency, voice transfer, and naturalness across five X-to-English tasks. Moreover, we demonstrate that our model can be adapted to support a new input language with less than 1000h of speech. We provide examples, model weights, inference code and we release a benchmark containing 45h of multilingual data for speech translation evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。