让扩散模型动画更稳,自动对齐人体姿态避免变形
TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation
- 测试时用普鲁克斯特校准人体姿态,提升对齐精度
- 在姿态不匹配场景下,保真度与一致性显著提升
- 无需重训练,可通用适配任意扩散模型动画系统
人像动画旨在从参考人像和目标动作视频生成人物运动视频。当前基于扩散模型的动画系统在身份迁移上精度高,但输出质量不稳定,仅当参考图像与目标姿态帧中的人体形态(尺度与旋转)对齐时才能达到最优效果。现实中此类形变错位普遍,严重制约实际应用。为此,我们提出测试时普鲁克斯特校准(TPC),通过为扩散模型提供校准后的参考图像,增强其对参考与目标图像间人体形状对应关系的理解,使系统在存在形变错位时仍保持高性能。该方法简单、通用,可无须额外训练即适配任意扩散模型动画系统,有效提升真实场景下的实用性。
原文摘要 · Abstract (English)
Human image animation aims to generate a human motion video from the inputs of a reference human image and a target motion video. Current diffusion-based image animation systems exhibit high precision in transferring human identity into targeted motion, yet they still exhibit irregular quality in their outputs. Their optimal precision is achieved only when the physical compositions (i.e., scale and rotation) of the human shapes in the reference image and target pose frame are aligned. In the absence of such alignment, there is a noticeable decline in fidelity and consistency. Especially, in real-world environments, this compositional misalignment commonly occurs, posing significant challenges to the practical usage of current systems. To this end, we propose Test-time Procrustes Calibration (TPC), which enhances the robustness of diffusion-based image animation systems by maintaining optimal performance even when faced with compositional misalignment, effectively addressing real-world scenarios. The TPC provides a calibrated reference image for the diffusion model, enhancing its capability to understand the correspondence between human shapes in the reference and target images. Our method is simple and can be applied to any diffusion-based image animation system in a model-agnostic manner, improving the effectiveness at test time without additional training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。