分层解耦面部动作,实现高保真实时可控人脸重演。
PortraitDirector: A Hierarchical Disentanglement Framework for Controllable and Real-time Facial Reenactment

- 将面部运动分解为物理动作与情绪语义两层,分别建模控制。
- 支持512x512分辨率下20帧/秒、端到端800毫秒延迟的实时重演。
- 适合需要精细表情控制的虚拟人、影视特效等场景使用。
现有面部重演方法在表现力与细粒度控制间存在权衡。整体模型常牺牲控制性以换取表现力,而注重控制的方法则难以保证真实感与有效解耦。本文提出PortraitDirector,将面部运动视为分层组合任务,通过层次化运动解耦与重构策略,将面部运动拆分为:空间层(含全局头部姿态与局部分离的面部表情)与语义层(全局情绪)。空间层中,局部表情从裁剪区域提取,并通过基于信息瓶颈的情绪过滤模块去除情感线索;语义层生成全局情绪表示。解耦成分再重组为表达性运动隐变量。通过扩散蒸馏、因果注意力与VAE加速等优化,实现单张5090显卡上512×512分辨率下20帧/秒、端到端800毫秒延迟的实时重演。
原文摘要 · Abstract (English)
Existing facial reenactment methods struggle with a trade-off between expressiveness and fine-grained controllability. Holistic facial reenactment models often sacrifice granular control for expressiveness, while methods designed for control may struggle with fidelity and robust disentanglement. Instead of treating facial motion as a monolithic signal, we explore an alternative compositional perspective. In this paper, we introduce PortraitDirector, a novel framework that formulates face reenactment as a hierarchical composition task, achieving high-fidelity and controllable results. We employ a Hierarchical Motion Disentanglement and Composition strategy, deconstructing facial motion into a Spatial Layer for physical movements and a Semantic Layer for emotional content. The Spatial Layer comprises: (i) global head pose, managed via a dedicated representation and injection pathway; (ii) spatially separated local facial expressions, distilled from cropped facial regions and purged of emotional cues via Emotion-Filtering Module leveraging an information bottleneck. The Semantic Layer contains a derived global emotion. The disentangled components are then recomposed into an expressive motion latent. Furthermore, we engineer the framework for real-time performance through a suite of optimizations, including diffusion distillation, causal attention and VAE acceleration. PortraitDirector achieves streaming, high-fidelity, controllable 512 x 512 face reenactment at 20 FPS with a end-to-end 800 ms latency on a single 5090 GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。