用上下文学习让任意角色图像自动动起来,效果更自然通用。
DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning
- 把动作控制看作上下文学习,统一处理外观与动态信息
- 自生成伪数据对,实现无需姿态先验的端到端动画生成
- 支持各种角色和动作,适合做通用角色动画开发
角色图像动画旨在通过将驱动序列中的运动迁移到静态参考图像上,生成高保真视频。尽管近期取得进展,现有方法仍面临两大根本挑战:(1) 动作注入策略不佳,导致身份保持与运动一致性之间存在权衡,表现为“跷跷板”效应;(2) 过度依赖显式姿态先验(如骨骼),难以捕捉复杂动态,限制了对任意非人形角色的泛化能力。为此,我们提出 DreamActor-M2,一个通用动画框架,将动作条件建模重构为上下文学习问题。该方法采用两阶段范式:首先,通过融合参考外观与运动线索到统一潜在空间,利用基础模型的生成先验,联合推理空间身份与时间动态;其次,引入自启动数据合成流程,构建伪跨角色训练对,实现从姿态依赖控制到直接端到端RGB驱动动画的平滑过渡。该策略显著提升在多样角色与运动场景下的泛化能力。为进一步推动评估,我们还构建了 AW Bench,一个涵盖广泛角色类型与运动场景的综合性基准。大量实验证明,DreamActor-M2 达到当前最优性能,兼具优异视觉保真度与强跨域泛化能力。
原文摘要 · Abstract (English)
Character image animation aims to synthesize high-fidelity videos by transferring motion from a driving sequence to a static reference image. Despite recent advancements, existing methods suffer from two fundamental challenges: (1) suboptimal motion injection strategies that lead to a trade-off between identity preservation and motion consistency, manifesting as a "see-saw", and (2) an over-reliance on explicit pose priors (e.g., skeletons), which inadequately capture intricate dynamics and hinder generalization to arbitrary, non-humanoid characters. To address these challenges, we present DreamActor-M2, a universal animation framework that reimagines motion conditioning as an in-context learning problem. Our approach follows a two-stage paradigm. First, we bridge the input modality gap by fusing reference appearance and motion cues into a unified latent space, enabling the model to jointly reason about spatial identity and temporal dynamics by leveraging the generative prior of foundational models. Second, we introduce a self-bootstrapped data synthesis pipeline that curates pseudo cross-identity training pairs, facilitating a seamless transition from pose-dependent control to direct, end-to-end RGB-driven animation. This strategy significantly enhances generalization across diverse characters and motion scenarios. To facilitate comprehensive evaluation, we further introduce AW Bench, a versatile benchmark encompassing a wide spectrum of characters types and motion scenarios. Extensive experiments demonstrate that DreamActor-M2 achieves state-of-the-art performance, delivering superior visual fidelity and robust cross-domain generalization. Project Page: https://grisoon.github.io/DreamActor-M2/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。