用自然语言描述生成带属性的语音嵌入分布,让语音合成更可控。
ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions

- 输入自然语言描述,生成对应语音嵌入的概率分布
- 在真实语音嵌入上实现高匹配率,属性分类准确率达90%以上
- 适合语音合成、变声等需要精准声线控制的应用
说话人嵌入(x-vectors)广泛用于表示说话人身份和相关属性,但现有提取器多为描述性而非生成性:将一段语音映射为x-vector后用于下游任务。我们提出ProPS(Prompted Profile Synthesis),一个基于自然语言提示(如“30岁男性,带有印度口音”)生成说话人嵌入分布的框架。ProPS将人工撰写的语音特征描述转换为句向量,通过在大规模数据集上训练的混合密度网络,预测x-vector空间中的高斯混合模型。模型通过最大化真实说话人嵌入与请求特征的匹配概率进行训练,并在保留的x-vectors上以负对数似然评估,同时在采样的合成x-vectors上测试属性分类准确率。实验表明,ProPS能生成符合指定特征(年龄、性别、口音、语调特征)的分布,且生成的嵌入保留了目标属性。该设计使语音合成系统(如文本转语音TTS或语音转换VC)具备可控的声线合成能力,同时保持生成分布与真实数据结构一致。
原文摘要 · Abstract (English)
Speaker embeddings, or x-vectors, are widely used to represent speaker identity and speaker-related attributes, but existing embedding extractors are typically descriptive rather than generative: they map an observed speech segment to an x-vector, which is then used for downstream applications. We introduce ProPS, Prompted Profile Synthesis, a framework for generating distributions of speaker embeddings conditioned on natural language prompts such as "a thirties male speaker with an Indian accent". ProPS converts human-written profile descriptions into sentence embeddings and uses a mixture density network trained on a large-scale dataset to predict a Gaussian mixture model in the x-vector space. The model is trained by maximizing the likelihood that real speaker embeddings match the requested profile, and its generated distributions are evaluated by negative log-likelihood on held-out x-vectors and by attribute classification accuracies on sampled synthetic x-vectors. Experiments show that ProPS produces profile-conditioned distributions and generates x-vectors that preserve requested speaker attributes such as age, gender, accent, and prosodic characteristics. This design enables controllable speaker-profile synthesis for speech generation systems like Text-To-Speech (TTS) or Voice Conversion (VC) while anchoring generated distributions in observed speaker-embedding structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。