实现低延迟多语言语音翻译,支持多人对话中说话人一致的实时合成。
X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System

- 分模块级联处理:流式语音识别+机器翻译+提示条件语音合成
- 通过增量段确认和在线说话人提示管理,提升长对话稳定性
- 开源系统,适合研究部署实用语音翻译的团队使用
实时语音到语音翻译(S2ST)系统需在翻译质量、延迟、语音自然度和说话人一致性之间取得平衡。现有公开S2ST系统已实现直接、多语言、流式与富有表现力的建模,而专有产品和API则逐步向用户提供实时翻译能力。然而,在开放可复现系统中,长时序、多说话人对话场景仍具挑战:部分语音识别结果不稳定,发言切换边界模糊,且目标语音需以合适说话人提示生成。本文提出X-Translator,一种低成本、模块化级联式的S2ST系统,通过会话级运行时控制器整合流式语音识别、机器翻译与提示条件语音合成。系统采用增量段确认机制,将不稳定的语音识别流转化为可翻译单元,并通过在线说话人提示管理器,将源语音段绑定至特定说话人声线进行合成。我们在OpenSTBench上评估了翻译质量、语音质量与延迟,对比了专有API作为行为基线,测量了长对话语音稳定性,评估了多说话人场景下的说话人保持效果,以及多语言翻译性能。X-Translator为理解面向部署的S2ST实际权衡提供了一个开源平台。代码与演示见https://github.com/zhaoyx239/X-Translator。
原文摘要 · Abstract (English)
Real-time speech-to-speech translation (S2ST) systems must balance translation quality, latency, speech naturalness, and speaker consistency. Publicly documented S2ST systems have advanced direct, multilingual, streaming, and expressive modeling, while proprietary products and APIs increasingly expose real-time translation capabilities to users. However, practical deployment remains challenging for open and reproducible systems, especially in long-form and multi-speaker conversations where partial ASR hypotheses are unstable, turn boundaries are ambiguous, and target speech must be generated with an appropriate speaker prompt. We present X-Translator, a low-cost modular cascaded S2ST system that combines streaming ASR, machine translation, and prompt-conditioned TTS through a session-level runtime controller. The system uses incremental segment commitment to convert unstable ASR streams into translation-ready units, and an online speaker prompt manager to bind source speech spans to speaker-specific voice prompts for synthesis. We evaluate translation, speech quality, and latency with OpenSTBench, compare against proprietary speech translation APIs as behavioral baselines, measure long-form voice stability, evaluate speaker preservation in multi-speaker conversations, and assess multilingual translation quality. X-Translator provides an open platform for understanding the practical trade-offs of deployment-oriented S2ST. Code and demo are available at https://github.com/zhaoyx239/X-Translator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。