用轻量微调实现高保真人体动画，支持高清无缝扩展。
UniAnimate-DiT: Human Image Animation with Large-Scale Video Diffusion Transformer
- 仅微调少量参数，通过LoRA保持原模型生成能力
- 480p训练后可直接推断至720p，效果自然连贯
- 融合参考图像姿态信息，提升动作对齐精度
本文提出UniAnimate-DiT，利用开源Wan2.1模型的强大生成能力，实现一致的人体图像动画。为保留原模型性能，采用低秩适配（LoRA）技术微调极少数参数，显著降低训练内存开销。设计轻量级3D卷积堆叠姿态编码器以捕捉驱动姿态的运动信息，并通过简单拼接操作融合参考外观特征，同时引入参考图像的姿态信息以增强动作对齐效果。实验表明，该方法生成的动画在视觉质量和时间一致性上均表现优异。模型在480p（832x480）视频上训练，推理时可无缝扩展至720P（1280x720），展现强大泛化能力。训练与推理代码已公开于https://github.com/ali-vilab/UniAnimate-DiT。
原文摘要 · Abstract (English)
This report presents UniAnimate-DiT, an advanced project that leverages the cutting-edge and powerful capabilities of the open-source Wan2.1 model for consistent human image animation. Specifically, to preserve the robust generative capabilities of the original Wan2.1 model, we implement Low-Rank Adaptation (LoRA) technique to fine-tune a minimal set of parameters, significantly reducing training memory overhead. A lightweight pose encoder consisting of multiple stacked 3D convolutional layers is designed to encode motion information of driving poses. Furthermore, we adopt a simple concatenation operation to integrate the reference appearance into the model and incorporate the pose information of the reference image for enhanced pose alignment. Experimental results show that our approach achieves visually appearing and temporally consistent high-fidelity animations. Trained on 480p (832x480) videos, UniAnimate-DiT demonstrates strong generalization capabilities to seamlessly upscale to 720P (1280x720) during inference. The training and inference code is publicly available at https://github.com/ali-vilab/UniAnimate-DiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。