arXiv:2605.17583cs.CV2026-05

让语音合成精准执行复杂指令,实现情感与风格的精确控制。

AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech

论文配图:AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
图 1 · 摘自论文原文
  • 用多智能体闭环框架分离说话人与情感特征,减少干扰
  • 通过声学原型库检索并融合表达锚点,生成连续控制信号
  • 引入反馈机制优化语音强度和语义一致性,适合高阶语音应用

现有文本到语音(TTS)模型虽具高表现力,但对复合指令的细粒度控制仍面临文本意图与连续声学实现之间的结构不匹配问题。受人类认知解耦启发,我们提出 AgentSteerTTS——一个用于复合指令忠实表达控制的多智能体闭环框架。首先,对抗解耦智能体通过泄漏抑制正则化学习可分离的身份与情感-韵律子空间,降低说话人-情感泄露。其次,双流锚定控制器利用大规模声学原型库实现抽象意图定位:检索智能体选取表达锚点,合成智能体通过门控注意力将其融合为连续控制向量。最后,快慢反馈智能体通过潜在梯度修正优化输出强度,并利用高层感知评判解决语义-声学不一致问题。在复合指令基准数据集及公开测试集上的实验表明,AgentSteerTTS显著优于基线方法,验证了该框架的有效性。

原文摘要 · Abstract (English)

While existing text-to-speech (TTS) models exhibit high expressiveness, fine-grained control over composite instructions remains challenging due to the structural mismatch between discrete textual intents and continuous acoustic realizations. Inspired by human cognitive decoupling, we introduce AgentSteerTTS, a multi-agent closed-loop framework designed for intent-faithful expressive control of composite instructions. First, in our framework, an adversarial disentanglement agent mitigates speaker-emotion leakage by learning separable identity and emotion-prosody subspaces with leakage-suppressing regularization. Next, a Dual-Stream Anchoring Controller grounds abstract intents using a large-scale acoustic prototype library: a Retrieval Agent selects expressive anchors, while a Synthesis Agent fuses them into continuous control vectors via gated attention. Finally, a Fast-Slow Feedback Agent refines output intensity through latent gradient correction and resolves semantic-acoustic mismatches using high-level perceptual critique. Experiments on a composite-instruction benchmark and public test sets show that AgentSteerTTS yields consistent and significant improvements to the baselines, demonstrating the effectiveness of the proposed method.

语音合成多智能体指令控制情感语音

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。