通过解耦姿态信息,实现更自然的人像可控动画生成。
DisPose: Disentangling Pose Guidance for Controllable Human Image Animation
- 将稀疏骨骼姿态分解为运动场与关键点对应关系
- 在不同体型下仍保持高质量动画一致性
- 可直接接入现有模型,无需重新训练
可控人像动画旨在利用驱动视频从参考图像生成视频。由于稀疏引导(如骨骼姿态)提供的控制信号有限,近期工作尝试引入额外的密集条件(如深度图)以确保动作对齐。然而,当参考角色与驱动视频中人体形态差异显著时,此类严格密集引导会损害生成视频质量。本文提出DisPose,无需额外密集输入即可挖掘更具泛化性与有效性的控制信号,将人像动画中的稀疏骨骼姿态解耦为运动场引导与关键点对应关系。具体而言,我们从稀疏运动场与参考图像生成稠密运动场,提供区域级密集引导的同时保持稀疏姿态控制的泛化能力;同时,从参考图像中提取对应姿态关键点的扩散特征,并将其转移至目标姿态以保留身份信息。为无缝集成至现有模型,我们提出一种即插即用的混合ControlNet,提升生成视频的质量与一致性,同时冻结原有模型参数。大量定性与定量实验表明,DisPose优于当前方法。项目页面:https://github.com/lihxxx/DisPose。
原文摘要 · Abstract (English)
Controllable human image animation aims to generate videos from reference images using driving videos. Due to the limited control signals provided by sparse guidance (e.g., skeleton pose), recent works have attempted to introduce additional dense conditions (e.g., depth map) to ensure motion alignment. However, such strict dense guidance impairs the quality of the generated video when the body shape of the reference character differs significantly from that of the driving video. In this paper, we present DisPose to mine more generalizable and effective control signals without additional dense input, which disentangles the sparse skeleton pose in human image animation into motion field guidance and keypoint correspondence. Specifically, we generate a dense motion field from a sparse motion field and the reference image, which provides region-level dense guidance while maintaining the generalization of the sparse pose control. We also extract diffusion features corresponding to pose keypoints from the reference image, and then these point features are transferred to the target pose to provide distinct identity information. To seamlessly integrate into existing models, we propose a plug-and-play hybrid ControlNet that improves the quality and consistency of generated videos while freezing the existing model parameters. Extensive qualitative and quantitative experiments demonstrate the superiority of DisPose compared to current methods. Project page: \href{https://github.com/lihxxx/DisPose}{https://github.com/lihxxx/DisPose}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。