解决人物动画中姿态错位和面部失真问题,保持身份一致。
StableAnimator++: Overcoming Pose Misalignment and Face Distortion for Human Image Animation
- 通过可学习层预测相似变换矩阵,对齐参考图与驱动姿态。
- 引入分布感知身份适配器,减少时间层干扰,提升身份一致性。
- 在去噪过程中融合哈密顿-雅可比-贝尔曼优化,增强面部细节真实感。
当前基于扩散模型的人像动画常因参考图像与驱动视频在体型或姿势上差异大而难以保持身份一致性。我们提出StableAnimator++,首个具备可学习姿态对齐能力的身份保持视频扩散框架,无需后处理即可生成高质量视频。该框架基于视频扩散模型,设计了训练与推理阶段的专用模块:首先利用奇异值分解(SVD)引导,通过可学习层预测参考图与驱动姿态间的相似变换矩阵,实现姿态对齐;接着使用现成编码器提取图像与面部嵌入,并通过全局内容感知面部编码器优化面部特征;为进一步保障身份一致性,提出分布感知身份适配器,在抑制时序层干扰的同时实现分布对齐。推理阶段,创新性地将基于哈密顿-雅可比-贝尔曼(HJB)的面部优化嵌入去噪过程,引导扩散轨迹以提升面部保真度。在多个基准测试上,StableAnimator++在定性和定量评估中均表现优异。
原文摘要 · Abstract (English)
Current diffusion models for human image animation often struggle to maintain identity (ID) consistency, especially when the reference image and driving video differ significantly in body size or position. We introduce StableAnimator++, the first ID-preserving video diffusion framework with learnable pose alignment, capable of generating high-quality videos conditioned on a reference image and a pose sequence without any post-processing. Building upon a video diffusion model, StableAnimator++ contains carefully designed modules for both training and inference, striving for identity consistency. In particular, StableAnimator++ first uses learnable layers to predict the similarity transformation matrices between the reference image and the driven poses via injecting guidance from Singular Value Decomposition (SVD). These matrices align the driven poses with the reference image, mitigating misalignment to a great extent. StableAnimator++ then computes image and face embeddings using off-the-shelf encoders, refining the face embeddings via a global content-aware Face Encoder. To further maintain ID, we introduce a distribution-aware ID Adapter that counteracts interference caused by temporal layers while preserving ID via distribution alignment. During the inference stage, we propose a novel Hamilton-Jacobi-Bellman (HJB) based face optimization integrated into the denoising process, guiding the diffusion trajectory for enhanced facial fidelity. Experiments on benchmarks show the effectiveness of StableAnimator++ both qualitatively and quantitatively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。