开源系统实现多语言翻译与语音克隆，支持实时通话与广播应用。
Open-Source System for Multilingual Translation and Cloned Speech Synthesis
- 用Whisper+VAD识别语音，双LLM分工处理句子分割与翻译。
- 语音克隆模块保持原声自然度，延迟低、准确率高。
- 适合远程会议、公共广播等场景，可本地或云端部署。
我们提出一个开源系统，用于多语言翻译与语音再生，解决跨语言交流与无障碍访问难题。系统采用Whisper进行语音识别并结合语音活动检测（VAD）定位说话时段，随后通过大型语言模型（LLMs）流水线处理：首个LLM将语音分段为完整句子，第二个则执行多语言翻译。在语音再生环节，系统使用具备语音克隆能力的文本到语音（TTS）模块，还原原始说话人音色，确保自然度与身份一致性。系统组件可本地运行或通过API调用，支持低成本部署，适用于Zoom会议中的实时多语言翻译、公共广播的语音再生，以及个人设备的蓝牙多语言播放。通过保留说话人声音特征，系统在翻译与再生任务中均提供无缝沉浸式体验。本文提供了详细的性能分析，包括延迟和词准确率，验证其在真实多语言场景中实现包容性、可扩展通信解决方案的潜力。
原文摘要 · Abstract (English)
We present an open-source system designed for multilingual translation and speech regeneration, addressing challenges in communication and accessibility across diverse linguistic contexts. The system integrates Whisper for speech recognition with Voice Activity Detection (VAD) to identify speaking intervals, followed by a pipeline of Large Language Models (LLMs). For multilingual applications, the first LLM segments speech into coherent, complete sentences, which a second LLM then translates. For speech regeneration, the system uses a text-to-speech (TTS) module with voice cloning capabilities to replicate the original speaker's voice, maintaining naturalness and speaker identity. The system's open-source components can operate locally or via APIs, offering cost-effective deployment across various use cases. These include real-time multilingual translation in Zoom sessions, speech regeneration for public broadcasts, and Bluetooth-enabled multilingual playback through personal devices. By preserving the speaker's voice, the system ensures a seamless and immersive experience, whether translating or regenerating speech. This open-source project is shared with the community to foster innovation and accessibility. We provide a detailed system performance analysis, including latency and word accuracy, demonstrating its potential to enable inclusive, adaptable communication solutions in real-world multilingual scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。