仅加一层零初始化线性层，即可让文本生成音视频模型实现语音克隆。
Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
- 在音频主干上加单层零初始化线性层，通过短参考音频实现语音克隆。
- 在674个说话人-文本对上，语音编码相似度显著优于5个主流基线。
- 可独立运行音频路径，推理速度提升30倍，适合实时语音克隆场景。
文本到音视频（T2AV）生成模型从文本描述生成视频和配乐，但无法控制输出语音的说话人。我们证明，仅需在基础T2AV模型的音频主干上添加一个零初始化的线性层，经过较短微调周期，并在推理时引入一段短参考音频，即可实现语音克隆。参考音频通过两种互补信号注入：其扩散潜变量被前置到音频流中，全局说话人嵌入则调制目标音频的标记。在包含30个说话人、共674个说话人-文本对的基准上，与五个强基线对比，我们的5B模型在三个独立验证网络（ECAPA-TDNN、WavLM-SV、Resemblyzer）上均达到最高的说话人编码相似度（SECS），统计显著更优。该架构的副产品是：音频路径可在推理时独立运行，相较完整音视频扩散循环提速约30倍，同时保持语音克隆能力。
原文摘要 · Abstract (English)
Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。