arXiv:2501.06467cs.CL2025-01中稿 · Information Fusion…被引 14

用历史对话知识增强表达性语音合成,让机器更懂情感回应。

Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis

  • 从存储对话中检索语义与风格相似的片段,构建双模态数据库。
  • 通过多粒度图结构聚合当前与历史对话的风格信息,提升表达力。
  • 在DailyTalk数据集上验证,合成语音更自然、更具共情感。

对话语音合成(CSS)旨在以当前对话历史为参考,生成与对话风格一致的富有表现力语音。与当前对话(CD)不同,存储对话(SD)包含用户-代理交互早期阶段保留的对话片段,其中蕴含与当前场景相似的风格表达知识,对生成有同理心的反馈至关重要。然而,现有研究忽略了这一方面。为此,本文提出一种新型检索增强式对话知识聚合方法RADKA-CSS,包含三个核心组件:1)构建包含文本与音频样本的存储对话语义-风格数据库(SDSSD),设计多属性检索机制,匹配CD的语义与风格向量,精准检索最相似的存储对话;2)采用多粒度图结构编码对话,并引入多源风格知识聚合机制,有效融合来自CD与SD的风格信息;3)将聚合后的风格知识输入语音合成器,指导生成符合对话风格的表达性语音。基于基准数据集DailyTalk的全面实验表明,RADKA-CSS在客观与主观评估中均优于基线模型,显著提升表达力呈现效果。代码与音频样例见:https://github.com/Coder-jzq/RADKA-CSS。

原文摘要 · Abstract (English)

Conversational speech synthesis (CSS) aims to take the current dialogue (CD) history as a reference to synthesize expressive speech that aligns with the conversational style. Unlike CD, stored dialogue (SD) contains preserved dialogue fragments from earlier stages of user-agent interaction, which include style expression knowledge relevant to scenarios similar to those in CD. Note that this knowledge plays a significant role in enabling the agent to synthesize expressive conversational speech that generates empathetic feedback. However, prior research has overlooked this aspect. To address this issue, we propose a novel Retrieval-Augmented Dialogue Knowledge Aggregation scheme for expressive CSS, termed RADKA-CSS, which includes three main components: 1) To effectively retrieve dialogues from SD that are similar to CD in terms of both semantic and style. First, we build a stored dialogue semantic-style database (SDSSD) which includes the text and audio samples. Then, we design a multi-attribute retrieval scheme to match the dialogue semantic and style vectors of the CD with the stored dialogue semantic and style vectors in the SDSSD, retrieving the most similar dialogues. 2) To effectively utilize the style knowledge from CD and SD, we propose adopting the multi-granularity graph structure to encode the dialogue and introducing a multi-source style knowledge aggregation mechanism. 3) Finally, the aggregated style knowledge are fed into the speech synthesizer to help the agent synthesize expressive speech that aligns with the conversational style. We conducted a comprehensive and in-depth experiment based on the DailyTalk dataset, which is a benchmarking dataset for the CSS task. Both objective and subjective evaluations demonstrate that RADKA-CSS outperforms baseline models in expressiveness rendering. Code and audio samples can be found at: https://github.com/Coder-jzq/RADKA-CSS.

语音合成对话系统风格迁移知识聚合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。