无需参考编码器，用梯度优化从语音合成模型中提取音色向量。
Extracting Voice Styles from Frozen TTS Models via Gradient-Based Inverse Optimization
- 通过梯度下降反向优化风格向量，固定模型权重只调风格向量。
- 在两个语料库154名说话人上，相似度提升至0.413（原0.132）。
- 适合想自定义音色但无参考编码器的用户，可复现他人声线。
一些文本到语音系统发布时仅包含合成模型和预设风格向量，但不提供将音频转为向量的参考编码器。尽管模型仍接受风格向量输入，但用户无法生成自己的风格向量。本文提出直接反向求解该输入：通过梯度下降逆推释放管道，所有权重保持冻结，仅优化风格向量，目标是匹配单段录音的时间池化WavLM统计特征。由于目标函数舍弃时间轴，合成文本可与原始录音不同，因此无需字幕或对齐信息。在两个语料库共154名说话人上，ECAPA-TDNN相似度由0.132升至0.413，ResNet相似度由0.099升至0.401，每位说话人均有提升；在等错误率点，验证器对恢复音色的认可率达53%，而原始预设仅为1%。
原文摘要 · Abstract (English)
Some text-to-speech systems ship a synthesis model and preset style vectors but not the reference encoder that turns audio into such a vector. The model still accepts a style vector; a user with a voice of their own cannot produce one. We solve for that input directly, inverting the released pipeline by gradient descent: every weight stays frozen and only the style vector is optimized, against time-pooled WavLM statistics of one recording. Because the objective discards the time axis, the synthesized text may differ from the recording, so no transcript and no alignment are needed. On 154 speakers from two corpora, ECAPA-TDNN similarity rises from 0.132 to 0.413 and ResNet from 0.099 to 0.401, improving for every speaker; a verifier at its equal-error point accepts 53% of the recovered voices as the target, against 1% for the presets they start from.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。