用法向图流操控注意力,让衣服动画更自然无噪点。
FloAt: Flow Warping of Self-Attention for Clothing Animation Generation
- 通过法向图流重构自注意力图,实现无训练控制
- 生成动画质量优于所有基线,背景闪烁显著减少
- 适合服装动画生成、影视特效等需要细腻动态的场景
我们提出一种基于扩散模型的方法FloAtControlNet,用于生成包含人物服装动画的动态影像(cinemagraphs)。针对连衣裙、裙子和裤子等服装,输入为描述服装类型与纹理(如豹纹、条纹或纯色)的文本提示,以及一组表征期望动作的法向图序列。方法核心是基于法向图条件的ControlNet,在无需训练的设定下运行。关键观察是:底层动画信息嵌入于法向图的运动流中。我们利用该流对特定层和帧的自注意力图进行重计算,将其表示为当前帧与前一帧自注意力图的线性组合,并以两帧法向图流进行空间扭曲。实验表明,此操作显著提升服装动画质量,使其更自然且有效抑制背景伪影。在视觉效果与用户评估上均超越所有基线;同时在输入法向图序列与输出图像生成的法向图序列之间,我们的方法在RMSE和PSNR指标上也全面领先。此外,我们发现传统视觉质量指标(如LPIPS、SSIM、CLIP分数)未必能捕捉服装细微动态,不适用于此类任务评估。
原文摘要 · Abstract (English)
We propose a diffusion model-based approach, FloAtControlNet to generate cinemagraphs composed of animations of human clothing. We focus on human clothing like dresses, skirts and pants. The input to our model is a text prompt depicting the type of clothing and the texture of clothing like leopard, striped, or plain, and a sequence of normal maps that capture the underlying animation that we desire in the output. The backbone of our method is a normal-map conditioned ControlNet which is operated in a training-free regime. The key observation is that the underlying animation is embedded in the flow of the normal maps. We utilize the flow thus obtained to manipulate the self-attention maps of appropriate layers. Specifically, the self-attention maps of a particular layer and frame are recomputed as a linear combination of itself and the self-attention maps of the same layer and the previous frame, warped by the flow on the normal maps of the two frames. We show that manipulating the self-attention maps greatly enhances the quality of the clothing animation, making it look more natural as well as suppressing the background artifacts. Through extensive experiments, we show that the method proposed beats all baselines both qualitatively in terms of visual results and user study. Specifically, our method is able to alleviate the background flickering that exists in other diffusion model-based baselines that we consider. In addition, we show that our method beats all baselines in terms of RMSE and PSNR computed using the input normal map sequences and the normal map sequences obtained from the output RGB frames. Further, we show that well-established evaluation metrics like LPIPS, SSIM, and CLIP scores that are generally for visual quality are not necessarily suitable for capturing the subtle motions in human clothing animations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。