用扩散Transformer实现多角色表情丰富的人像动画,解决表情失真和角色干扰问题。
FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers
- 通过隐式表达学习捕捉无身份依赖的面部动态
- 在跨角色重演和多角色场景中显著提升动画质量
- 适用于影视特效、虚拟主播等需要多角色表情同步的场景
从静态图像生成富有表现力的面部动画是一项挑战性任务。以往依赖显式几何先验(如面部关键点或3DMM)的方法在跨角色重演中常出现伪影,难以捕捉细微情绪。此外,现有方法缺乏对多角色动画的支持,不同个体的驱动特征容易相互干扰。为此,我们提出FantasyPortrait,一种基于扩散Transformer的框架,可在单角色与多角色场景下生成高保真、情绪丰富的动画。该方法引入表达增强学习策略,利用隐式表示捕捉无身份依赖的面部动态,提升细腻情绪渲染能力;针对多角色控制,设计掩码交叉注意力机制,确保表达生成独立且协调,有效防止特征干扰。为推动该领域研究,我们构建了Multi-Expr数据集和ExprBench基准。大量实验表明,FantasyPortrait在定量指标与定性评估上均显著优于现有方法,尤其在跨角色重演和多角色场景中表现突出。
原文摘要 · Abstract (English)
Producing expressive facial animations from static images is a challenging task. Prior methods relying on explicit geometric priors (e.g., facial landmarks or 3DMM) often suffer from artifacts in cross reenactment and struggle to capture subtle emotions. Furthermore, existing approaches lack support for multi-character animation, as driving features from different individuals frequently interfere with one another, complicating the task. To address these challenges, we propose FantasyPortrait, a diffusion transformer based framework capable of generating high-fidelity and emotion-rich animations for both single- and multi-character scenarios. Our method introduces an expression-augmented learning strategy that utilizes implicit representations to capture identity-agnostic facial dynamics, enhancing the model's ability to render fine-grained emotions. For multi-character control, we design a masked cross-attention mechanism that ensures independent yet coordinated expression generation, effectively preventing feature interference. To advance research in this area, we propose the Multi-Expr dataset and ExprBench, which are specifically designed datasets and benchmarks for training and evaluating multi-character portrait animations. Extensive experiments demonstrate that FantasyPortrait significantly outperforms state-of-the-art methods in both quantitative metrics and qualitative evaluations, excelling particularly in challenging cross reenactment and multi-character contexts. Our project page is https://fantasy-amap.github.io/fantasy-portrait/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。