解决人脸配音中细节模糊问题,让生成角色更像真人。
Removing Averaging: Personalized Lip-Sync Driven Characters Based on Identity Adapter
- 用身份编码器和嵌入注入机制,保留说话人面部特征。
- 在两个数据集上提升5%身份一致性、2%图像相似度。
- 适合需要高保真人物形象的视频配音场景。
基于扩散模型的唇同步生成技术虽能实现精准口型同步,但常因“唇部平均化”现象丢失细微面部特征。该问题源于常用UNet结构通过交叉注意力与多尺度融合整合音频特征,难以保留生成人脸的精细唇部细节。为此,本文提出UnAvgLip方法:利用参考视频提取身份嵌入,生成高度忠实的面部序列并保持精确唇同步。方法包含两个核心组件:(1)身份感知模块,将面部嵌入对齐至音频条件;(2)身份交叉注意力模块,将面部嵌入注入生成过程,增强身份保持能力。大量实验表明,仅需适度训练与推理开销,UnAvgLip有效缓解唇部平均化问题,在HDTF与LRW两个基准数据集上,身份一致性提升5%,SSIM指标提升2%。
原文摘要 · Abstract (English)
Recent advances in diffusion-based lip-syncing generative models have demonstrated their ability to produce highly synchronized talking face videos for visual dubbing. Although these models excel at lip synchronization, they often struggle to maintain fine-grained control over facial details in generated images. In this work, we identify "lip averaging" phenomenon where the model fails to preserve subtle facial details when dubbing unseen in-the-wild videos. This issue arises because the commonly used UNet backbone primarily integrates audio features into visual representations in the latent space via cross-attention mechanisms and multi-scale fusion, but it struggles to retain fine-grained lip details in the generated faces. To address this issue, we propose UnAvgLip, which extracts identity embeddings from reference videos to generate highly faithful facial sequences while maintaining accurate lip synchronization. Specifically, our method comprises two primary components: (1) an Identity Perceiver module that encodes facial embeddings to align with conditioned audio features; and (2) an ID-CrossAttn module that injects facial embeddings into the generation process, enhancing model's capability of identity retention. Extensive experiments demonstrate that, at a modest training and inference cost, UnAvgLip effectively mitigates the "averaging" phenomenon in lip inpainting, significantly preserving unique facial characteristics while maintaining precise lip synchronization. Compared with the original approach, our method demonstrates significant improvements of 5% on the identity consistency metric and 2% on the SSIM metric across two benchmark datasets (HDTF and LRW).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。