让人脸动画随光照变化,实现高保真可调控的光影效果。
High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model
- 分离身份与光照姿态特征,分别在不同子空间中控制
- 生成视频在光照真实度和画质上超越现有方法
- 适合需要精细光影控制的人像动画应用
可重光照人脸动画旨在将静态参考人脸动画化,匹配驱动视频中的头部运动与表情,同时适应用户指定或参考的光照条件。现有方法无法实现真正可重光照的人脸动画,因其未分离并操控内在(身份与外观)与外在(姿态与光照)特征。本文提出光照可控视频扩散模型(LCVD),实现高保真、可重光照的人脸动画。通过预训练图像到视频扩散模型的特征空间,设计专用子空间区分两类特征:利用3D网格、姿态及光照渲染的阴影提示表示外在属性,参考图像代表内在属性。训练阶段,采用参考适配器将参考映射至内在特征子空间,阴影适配器将阴影提示映射至外在特征子空间。融合两子空间特征后,模型可精细控制生成动画中的光照、姿态与表情。大量评估表明,LCVD在光照真实度、图像质量与视频一致性方面均优于当前最优方法,为可重光照人脸动画树立了新基准。
原文摘要 · Abstract (English)
Relightable portrait animation aims to animate a static reference portrait to match the head movements and expressions of a driving video while adapting to user-specified or reference lighting conditions. Existing portrait animation methods fail to achieve relightable portraits because they do not separate and manipulate intrinsic (identity and appearance) and extrinsic (pose and lighting) features. In this paper, we present a Lighting Controllable Video Diffusion model (LCVD) for high-fidelity, relightable portrait animation. We address this limitation by distinguishing these feature types through dedicated subspaces within the feature space of a pre-trained image-to-video diffusion model. Specifically, we employ the 3D mesh, pose, and lighting-rendered shading hints of the portrait to represent the extrinsic attributes, while the reference represents the intrinsic attributes. In the training phase, we employ a reference adapter to map the reference into the intrinsic feature subspace and a shading adapter to map the shading hints into the extrinsic feature subspace. By merging features from these subspaces, the model achieves nuanced control over lighting, pose, and expression in generated animations. Extensive evaluations show that LCVD outperforms state-of-the-art methods in lighting realism, image quality, and video consistency, setting a new benchmark in relightable portrait animation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。