统一语音与文本提示的音色控制，用简单注意力机制实现高质量合成。
CAST-TTS: A Simple Cross-Attention Framework for Unified Timbre Control in TTS
- 用预训练编码器提取语音与文本特征，通过多阶段训练对齐到共享空间。
- 单一交叉注意力模块支持任一输入控制音色，合成质量媲美专用模型。
- 结构简洁，适合需要统一音色控制的工业级语音合成系统使用。
当前文语转换（TTS）系统通常采用独立模型分别处理语音提示和文本提示的音色控制。虽然统一两种控制信号于单一模型更具吸引力，但跨模态对齐难题常导致架构复杂、训练目标繁琐。为此，我们提出CAST-TTS，一种简洁有效的统一音色控制框架。通过预训练编码器分别提取语音提示与文本提示特征，采用多阶段训练策略在共享嵌入空间中高效对齐语音与投影后的文本表示。随后，仅需一个交叉注意力机制即可让模型根据任一输入控制音色。大量实验验证，统一的交叉注意力机制对高质量语音合成至关重要。CAST-TTS在统一架构下达到与专用单输入模型相当的性能。演示页面可访问：https://HiRookie9.github.io/CAST-TTS-Page。
原文摘要 · Abstract (English)
Current Text-to-Speech (TTS) systems typically use separate models for speech-prompted and text-prompted timbre control. While unifying both control signals into a single model is desirable, the challenge of cross-modal alignment often results in overly complex architectures and training objective. To address this challenge, we propose CAST-TTS, a simple yet effective framework for unified timbre control. Features are extracted from speech prompts and text prompts using pre-trained encoders. The multi-stage training strategy efficiently aligns the speech and projected text representations within a shared embedding space. A single cross-attention mechanism then allows the model to use either of these representations to control the timbre. Extensive experiments validate that the unified cross-attention mechanism is critical for achieving high-quality synthesis. CAST-TTS achieves performance comparable to specialized single-input models while operating within a unified architecture. The demo page can be accessed at https://HiRookie9.github.io/CAST-TTS-Page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。