用表情关键点驱动,实现高效可控的自由风格人脸动画。
Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation
- 以关键点为运动信号,提升表情对齐与身份保持。
- 引入渐进生成与泰勒缓存,速度提升2.6倍且无质量损失。
- 支持真人、卡通、雕塑等多类型人物,适合内容创作者使用。
我们提出 Follow-Your-Emoji-Faster,一种基于扩散模型的自由风格人脸动画框架,由面部关键点驱动。核心挑战包括保持参考肖像身份、准确转移目标表情,以及确保长时间序列的时序一致性与生成效率。为此,我们在 Stable Diffusion 基础上引入两个关键组件:表达感知的关键点作为显式运动信号,改善运动对齐,支持夸张表情并减少身份泄露;细粒度面部损失结合表情与面部掩码,更好捕捉细微表情并忠实保留参考外观。该方法可实现多样肖像(真实人脸、卡通、雕塑、动物)的可控且富有表现力的动画。针对扩散模型生成长序列动画效率低的问题,我们提出渐进生成策略与泰勒插值缓存,实现2.6倍无损加速。最终,我们构建 EmojiBench++,包含多样化肖像、驱动视频和关键点序列,大量实验表明本方法在动画质量与可控性上均优于现有方案。代码、训练数据与基准将在 https://follow-your-emoji.github.io/ 公开。
原文摘要 · Abstract (English)
We present Follow-Your-Emoji-Faster, an efficient diffusion-based framework for freestyle portrait animation driven by facial landmarks. The main challenges in this task are preserving the identity of the reference portrait, accurately transferring target expressions, and maintaining long-term temporal consistency while ensuring generation efficiency. To address identity preservation and accurate expression retargeting, we enhance Stable Diffusion with two key components: a expression-aware landmarks as explicit motion signals, which improve motion alignment, support exaggerated expressions, and reduce identity leakage; and a fine-grained facial loss that leverages both expression and facial masks to better capture subtle expressions and faithfully preserve the reference appearance. With these components, our model supports controllable and expressive animation across diverse portrait types, including real faces, cartoons, sculptures, and animals. However, diffusion-based frameworks typically struggle to efficiently generate long-term stable animation results, which remains a core challenge in this task. To address this, we propose a progressive generation strategy for stable long-term animation, and introduce a Taylor-interpolated cache, achieving a 2.6X lossless acceleration. These two strategies ensure that our method produces high-quality results efficiently, making it user-friendly and accessible. Finally, we introduce EmojiBench++, a more comprehensive benchmark comprising diverse portraits, driving videos, and landmark sequences. Extensive evaluations on EmojiBench++ demonstrate that Follow-Your-Emoji-Faster achieves superior performance in both animation quality and controllability. The code, training dataset and benchmark will be found in https://follow-your-emoji.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。