用视频驱动单张人像,生成逼真表情与环境互动的动态影像。
X-Dyna: Expressive Dynamic Human Image Animation
- 基于扩散模型,通过动态适配器融合外观上下文信息。
- 零样本迁移下实现表情与动作精准传递,提升动画真实感。
- 适合影视特效、虚拟主播等需要高保真动态生成的场景。
我们提出X-Dyna,一种全新的零样本、基于扩散模型的人像动态生成方法,仅需一张人像图和一段驱动视频,即可生成包含面部表情与身体动作的逼真动态视频,并同步模拟人物与周围环境的自然交互。针对以往方法在动态细节还原上的不足,X-Dyna引入轻量级动态适配器(Dynamics-Adapter),将参考图像的外观上下文有效融入扩散主干的空间注意力机制中,同时保留运动模块对流畅、复杂动态细节的生成能力。此外,通过连接局部控制模块,模型可分离身份无关的面部表情,实现更精确的表情迁移。该框架能从多样化的真人与场景视频中学习物理性人体运动及自然场景动态,经全面定性与定量评估,性能超越现有最佳方法,生成高度逼真且富有表现力的动画。代码已开源:https://github.com/bytedance/X-Dyna。
原文摘要 · Abstract (English)
We introduce X-Dyna, a novel zero-shot, diffusion-based pipeline for animating a single human image using facial expressions and body movements derived from a driving video, that generates realistic, context-aware dynamics for both the subject and the surrounding environment. Building on prior approaches centered on human pose control, X-Dyna addresses key shortcomings causing the loss of dynamic details, enhancing the lifelike qualities of human video animations. At the core of our approach is the Dynamics-Adapter, a lightweight module that effectively integrates reference appearance context into the spatial attentions of the diffusion backbone while preserving the capacity of motion modules in synthesizing fluid and intricate dynamic details. Beyond body pose control, we connect a local control module with our model to capture identity-disentangled facial expressions, facilitating accurate expression transfer for enhanced realism in animated scenes. Together, these components form a unified framework capable of learning physical human motion and natural scene dynamics from a diverse blend of human and scene videos. Comprehensive qualitative and quantitative evaluations demonstrate that X-Dyna outperforms state-of-the-art methods, creating highly lifelike and expressive animations. The code is available at https://github.com/bytedance/X-Dyna.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。