无需文本的语音对语音翻译，让配音自然同步原声
Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing
- 用离散扩散模型实现时序对齐的语音单位翻译
- 生成语音与源语速一致，时长匹配率超90%
- 适合影视配音、实时字幕等需语音风格保留场景
本文提出一种跨语言配音系统，可在不依赖文本的情况下，将语音从一种语言翻译为另一种语言，同时保留语速、说话人身份和持续时间等关键特征。现有语音翻译方法虽翻译质量高，但常忽视语音模式传递，导致与源语音不匹配，难以用于配音。为此，我们设计了一种基于离散扩散的语音到语音单位翻译模型，并引入显式时长控制机制，实现时序对齐翻译。随后，利用条件流匹配模型，基于翻译后的语音单位和源说话人身份合成目标语音。此外，提出一种基于语音单位的语速自适应机制，使翻译输出语速与源语音保持一致，完全无需文本信息。大量实验表明，该框架生成的翻译结果自然流畅，且在时长和语速上高度匹配原始语音，同时具备竞争性的翻译性能。代码已开源：https://github.com/kaistmm/Dub-S2ST。
原文摘要 · Abstract (English)
This paper introduces a cross-lingual dubbing system that translates speech from one language to another while preserving key characteristics such as duration, speaker identity, and speaking speed. Despite the strong translation quality of existing speech translation approaches, they often overlook the transfer of speech patterns, leading to mismatches with source speech and limiting their suitability for dubbing applications. To address this, we propose a discrete diffusion-based speech-to-unit translation model with explicit duration control, enabling time-aligned translation. We then synthesize speech based on the translated units and source speaker's identity using a conditional flow matching model. Additionally, we introduce a unit-based speed adaptation mechanism that guides the translation model to produce speech at a rate consistent with the source, without relying on any text. Extensive experiments demonstrate that our framework generates natural and fluent translations that align with the original speech's duration and speaking pace, while achieving competitive translation performance. The code is available at https://github.com/kaistmm/Dub-S2ST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。