用上下文学习实现零样本语音转换，提升语调控制力
Prosody-Adaptable Audio Codecs for Zero-Shot Voice Conversion via In-Context Learning
- 在VALLE-X框架中引入上下文学习，实现无需训练的说话人适配
- 提出语调感知编码器（PACE），分离并优化语调特征，提升表达力
- 适合需要灵活语调调整的语音合成与跨说话人转换场景
最近离散音频编码器的进步显著提升了语音表征建模能力，而编码器语言模型则使零样本语音合成成为可能。受此启发，我们在VALLE-X框架内提出一种语音转换（VC）模型，利用其强大的上下文学习能力实现说话人自适应。为增强语调控制，我们引入语调感知音频编码器（PACE）模块，可分离并优化语调信息，减少其他因素干扰，提升表达丰富性与可控性。将PACE集成至VC模型后，可在保持说话人音色一致性的同时，实现更灵活的语调调节。实验结果表明，该方法在语调保留、音色一致性和整体自然度方面均优于基线系统。
原文摘要 · Abstract (English)
Recent advances in discrete audio codecs have significantly improved speech representation modeling, while codec language models have enabled in-context learning for zero-shot speech synthesis. Inspired by this, we propose a voice conversion (VC) model within the VALLE-X framework, leveraging its strong in-context learning capabilities for speaker adaptation. To enhance prosody control, we introduce a prosody-aware audio codec encoder (PACE) module, which isolates and refines prosody from other sources, improving expressiveness and control. By integrating PACE into our VC model, we achieve greater flexibility in prosody manipulation while preserving speaker timbre. Experimental evaluation results demonstrate that our approach outperforms baseline VC systems in prosody preservation, timbre consistency, and overall naturalness, surpassing baseline VC systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。