用新结构让扩散模型生成更自然的人体动画,保留文本语义还能精准控姿。
DynamiCtrl: Rethinking the Basic Structure and the Role of Text for High-quality Human Image Animation
- 统一图像与动作的潜在空间,用新层归一化实现跨块姿态控制
- 在自建数据集上达到0.166的最优LPIPS,生成质量显著提升
- 适合需要高保真角色动画和多层级控制的研究者与开发者
尽管扩散变换器(DiT)在视频生成中表现优异,但将其用于姿态引导的人体图像动画仍面临两大挑战:(a) 传统的基于U-Net的姿态控制方法对DiT主干不适用;(b) 去除文本引导会导致语义丢失与模型退化。为此,我们提出DynamiCtrl框架。首先,采用共享的VAE编码器处理人体图像与驱动姿态,将二者映射至同一潜在空间,保持姿态保真度,并在视频去噪中无需专家姿态编码器。其次,设计了新型的姿势自适应层归一化模块,通过视觉标记条件注入归一化后的姿态特征,实现对DiT各块的无缝且可扩展的姿态控制。此外,为克服移除文本的缺陷,引入“联合文本”范式,保留文本嵌入以提供全局语义上下文。通过全注意力机制,图像与姿态特征与文本特征对齐,增强语义一致性,利用预训练知识,实现多层次控制。实验表明,DynamiCtrl在基准数据集与自收集数据上均表现优越(如取得最佳LPIPS为0.166),展现出强大的角色控制能力与高质量合成效果。
原文摘要 · Abstract (English)
With diffusion transformer (DiT) excelling in video generation, its use in specific tasks has drawn increasing attention. However, adapting DiT for pose-guided human image animation faces two core challenges: (a) existing U-Net-based pose control methods may be suboptimal for the DiT backbone; and (b) removing text guidance, as in previous approaches, often leads to semantic loss and model degradation. To address these issues, we propose DynamiCtrl, a novel framework for human animation in video DiT architecture. Specifically, we use a shared VAE encoder for human images and driving poses, unifying them into a common latent space, maintaining pose fidelity, and eliminating the need for an expert pose encoder during video denoising. To integrate pose control into the DiT backbone effectively, we propose a novel Pose-adaptive Layer Norm model. It injects normalized pose features into the denoising process via conditioning on visual tokens, enabling seamless and scalable pose control across DiT blocks. Furthermore, to overcome the shortcomings of text removal, we introduce the "Joint-text" paradigm, which preserves the role of text embeddings to provide global semantic context. Through full-attention blocks, image and pose features are aligned with text features, enhancing semantic consistency, leveraging pretrained knowledge, and enabling multi-level control. Experiments verify the superiority of DynamiCtrl on benchmark and self-collected data (e.g., achieving the best LPIPS of 0.166), demonstrating strong character control and high-quality synthesis. The project page is available at https://gulucaptain.github.io/DynamiCtrl/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。