让角色动画手部更真实,还能抗干扰、防抖动。
RealisDance: Equip controllable character animation with realistic hands
- 用三种姿态融合+3D手部信息生成逼真手势
- 手部清晰度显著提升,复杂动作也不模糊
- 适合做高质量角色动画的开发者或创作者
可控角色动画能根据给定角色图像和姿态序列生成视频。现有方法在角色一致性上进步明显,但姿态控制仍不足:1)输入姿态损坏时生成失败;2)使用DWPose生成的手部模糊不真实;3)姿态不平滑导致视频抖动。本文提出RealisDance解决上述问题:自适应融合三种姿态,其中HaMeR提供精准3D与深度信息,使复杂手势也能真实生成;主UNet与姿态引导网络均引入时间注意力,从姿态条件层面平滑视频;训练中加入姿态打乱增强,进一步提升鲁棒性与流畅性。定性实验表明,RealisDance在手部质量上显著优于现有方法。
原文摘要 · Abstract (English)
Controllable character animation is an emerging task that generates character videos controlled by pose sequences from given character images. Although character consistency has made significant progress via reference UNet, another crucial factor, pose control, has not been well studied by existing methods yet, resulting in several issues: 1) The generation may fail when the input pose sequence is corrupted. 2) The hands generated using the DWPose sequence are blurry and unrealistic. 3) The generated video will be shaky if the pose sequence is not smooth enough. In this paper, we present RealisDance to handle all the above issues. RealisDance adaptively leverages three types of poses, avoiding failed generation caused by corrupted pose sequences. Among these pose types, HaMeR provides accurate 3D and depth information of hands, enabling RealisDance to generate realistic hands even for complex gestures. Besides using temporal attention in the main UNet, RealisDance also inserts temporal attention into the pose guidance network, smoothing the video from the pose condition aspect. Moreover, we introduce pose shuffle augmentation during training to further improve generation robustness and video smoothness. Qualitative experiments demonstrate the superiority of RealisDance over other existing methods, especially in hand quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。