通过分频建模实现更精准的3D虚拟人脸表情控制。
Toward Fine-Grained Facial Control in 3D Talking Head Generation
- 按面部运动频率分层建模,低频与高频区域分别处理。
- 在常见数据集上实现更准确的口型同步与减少面部抖动。
- 适合需要高真实感数字人交互的场景,如虚拟主播或客服。
音频驱动的虚拟人脸生成是数字人技术的核心,3D高斯点云渲染在实时生成高保真人脸方面表现优异。然而,精细面部动作的精确控制仍面临挑战,尤其是口型不同步和面部抖动问题,易引发恐怖谷效应。为此,本文提出细粒度3D高斯点云(FG-3DGS)框架,实现时序一致且高保真的虚拟人脸生成。方法引入频率感知解耦策略,根据运动特性对脸部区域进行建模:低频区域(如脸颊、鼻部、额头)由标准MLP联合建模,高频区域(如眼睛、嘴部)则通过专用网络结合面部掩码单独捕捉。预测的运动动态以高斯增量形式作用于静态高斯点,经逐帧相机参数驱动的光栅化器生成最终图像。此外,采用基于大规模音视频对预训练的高频精细化后处理对齐机制,提升逐帧生成质量,实现更精准的口型同步。在多个主流说话头生成数据集上的大量实验表明,该方法优于近期先进模型,能生成高保真、唇音同步的视频。
原文摘要 · Abstract (English)
Audio-driven talking head generation is a core component of digital avatars, and 3D Gaussian Splatting has shown strong performance in real-time rendering of high-fidelity talking heads. However, achieving precise control over fine-grained facial movements remains a significant challenge, particularly due to lip-synchronization inaccuracies and facial jitter, both of which can contribute to the uncanny valley effect. To address these challenges, we propose Fine-Grained 3D Gaussian Splatting (FG-3DGS), a novel framework that enables temporally consistent and high-fidelity talking head generation. Our method introduces a frequency-aware disentanglement strategy to explicitly model facial regions based on their motion characteristics. Low-frequency regions, such as the cheeks, nose, and forehead, are jointly modeled using a standard MLP, while high-frequency regions, including the eyes and mouth, are captured separately using a dedicated network guided by facial area masks. The predicted motion dynamics, represented as Gaussian deltas, are applied to the static Gaussians to generate the final head frames, which are rendered via a rasterizer using frame-specific camera parameters. Additionally, a high-frequency-refined post-rendering alignment mechanism, learned from large-scale audio-video pairs by a pretrained model, is incorporated to enhance per-frame generation and achieve more accurate lip synchronization. Extensive experiments on widely used datasets for talking head generation demonstrate that our method outperforms recent state-of-the-art approaches in producing high-fidelity, lip-synced talking head videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。