用文本生成同步的语音和人脸，无需真实音频。
Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis
- 用Wav2Vec2嵌入从文本生成语音与面部动作
- 两阶段训练提升音画对齐，保持说话人特征
- 适合做虚拟人、语音驱动动画的开发者
我们提出一种文本到对话脸合成框架，利用HierSpeech++的潜在语音表示。文本到向量模块从文本生成Wav2Vec2嵌入，联合控制语音与面部生成。为应对干净特征与TTS预测特征间的分布偏移，采用两阶段训练：先在Wav2Vec2嵌入上预训练，再在TTS输出上微调。该方法实现紧密的音画对齐，保留说话人身份，生成自然、有表现力的语音与同步的面部运动，且推理时无需真实音频。实验表明，基于TTS预测潜在特征的条件生成优于级联流水线，在唇形同步与视觉真实感方面均有提升。
原文摘要 · Abstract (English)
We propose a text-to-talking-face synthesis framework leveraging latent speech representations from HierSpeech++. A Text-to-Vec module generates Wav2Vec2 embeddings from text, which jointly condition speech and face generation. To handle distribution shifts between clean and TTS-predicted features, we adopt a two-stage training: pretraining on Wav2Vec2 embeddings and finetuning on TTS outputs. This enables tight audio-visual alignment, preserves speaker identity, and produces natural, expressive speech and synchronized facial motion without ground-truth audio at inference. Experiments show that conditioning on TTS-predicted latent features outperforms cascaded pipelines, improving both lip-sync and visual realism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。