用单语语音文本数据实现零样本语音翻译，不依赖成对语音数据
RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech
- 利用文本作语义桥梁合成翻译目标，无需平行语音对
- 德语转英语和西班牙语转英语分别达25.17和29.86的ASR-BLEU
- 保留原说话人音色，适合低资源语言的语音翻译
端到端语音到语音翻译（S2ST）系统常面临平行语音语料稀缺的数据瓶颈。为解决此问题，我们提出RosettaSpeech，一种仅使用单语语音-文本数据并辅以机器翻译监督的新型零样本框架。与依赖复杂级联伪标签的方法不同，本方法在训练中巧妙利用文本作为语义桥梁来合成翻译目标，从而无需平行语音对即可保持端到端推理流程。在CVSS-C基准上的实证评估表明，RosettaSpeech达到当前最优的零样本性能，相较领先基线显著提升：德语→英语的ASR-BLEU达25.17（相对增益+27%），西班牙语→英语达29.86（+14%）。关键的是，模型在未见过配对语音数据的情况下仍能有效保留源说话人音色。我们进一步分析了数据规模的影响，并验证了模型在多对一翻译中的能力，为扩展高质量S2ST至“文本丰富、语音稀少”的语言提供了可扩展解决方案。
原文摘要 · Abstract (English)
End-to-end speech-to-speech translation (S2ST) systems typically struggle with a critical data bottleneck: the scarcity of parallel speech-to-speech corpora. To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by machine translation supervision. Unlike prior works that rely on complex cascaded pseudo-labeling, our approach strategically utilizes text as a semantic bridge during training to synthesize translation targets, thereby eliminating the need for parallel speech pairs while maintaining a direct, end-to-end inference pipeline. Empirical evaluations on the CVSS-C benchmark demonstrate that RosettaSpeech achieves state-of-the-art zero-shot performance, surpassing leading baselines by significant margins - achieving ASR-BLEU scores of 25.17 for German-to-English (+27% relative gain) and 29.86 for Spanish-to-English (+14%). Crucially, our model effectively preserves the source speaker's voice without ever seeing paired speech data. We further analyze the impact of data scaling and demonstrate the model's capability in many-to-one translation, offering a scalable solution for extending high-quality S2ST to "text-rich, speech-poor" languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。