用动态神经高斯场实现逼真头部替换,解决表情不自然和背景融合差问题。
GSwap: Realistic Head Swapping with Dynamic Neural Gaussian Field
- 基于动态神经高斯场建模全身姿态,提升3D一致性。
- 在多个指标上超越现有方法,尤其在身份保留与运动连贯性上表现优异。
- 适合需要高质量人脸替换的影视制作与虚拟人应用。
我们提出GSwap,一种基于动态神经高斯肖像先验的稳定且逼真的视频头部替换系统,显著推动了面部与头部替换的技术水平。与依赖2D生成模型或3D可变形人脸模型(3DMM)的以往方法不同,本方法克服了其固有的局限性,包括较差的3D一致性、不自然的面部表情以及合成质量受限等问题。此外,现有技术在完整头部替换任务中因缺乏整体头部建模与无效背景融合,常导致可见伪影和错位。GSwap引入嵌入于全身SMPL-X表面的内在3D高斯特征场,将2D肖像视频升级为动态神经高斯场,确保高保真度、3D一致性的肖像渲染,同时保持自然的头-躯关系与流畅的运动动态。为支持训练,仅需少量参考图像即可将预训练2D肖像生成模型适配至源头域,实现高效领域迁移。此外,提出一种神经重渲染策略,无缝融合合成前景与原始背景,消除融合伪影并提升真实感。大量实验表明,GSwap在视觉质量、时间连贯性、身份保留与3D一致性等方面均优于现有方法。
原文摘要 · Abstract (English)
We present GSwap, a novel consistent and realistic video head-swapping system empowered by dynamic neural Gaussian portrait priors, which significantly advances the state of the art in face and head replacement. Unlike previous methods that rely primarily on 2D generative models or 3D Morphable Face Models (3DMM), our approach overcomes their inherent limitations, including poor 3D consistency, unnatural facial expressions, and restricted synthesis quality. Moreover, existing techniques struggle with full head-swapping tasks due to insufficient holistic head modeling and ineffective background blending, often resulting in visible artifacts and misalignments. To address these challenges, GSwap introduces an intrinsic 3D Gaussian feature field embedded within a full-body SMPL-X surface, effectively elevating 2D portrait videos into a dynamic neural Gaussian field. This innovation ensures high-fidelity, 3D-consistent portrait rendering while preserving natural head-torso relationships and seamless motion dynamics. To facilitate training, we adapt a pretrained 2D portrait generative model to the source head domain using only a few reference images, enabling efficient domain adaptation. Furthermore, we propose a neural re-rendering strategy that harmoniously integrates the synthesized foreground with the original background, eliminating blending artifacts and enhancing realism. Extensive experiments demonstrate that GSwap surpasses existing methods in multiple aspects, including visual quality, temporal coherence, identity preservation, and 3D consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。