arXiv:2501.16714cs.CVcs.AI2025-01被引 6

让视频生成模型只学动作不学外形,生成更符合描述的动作视频。

Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models

  • 用新方法分离动作与外观,仅通过动作LoRA调整时间注意力。
  • 生成视频动作更贴合参考视频,外观更符合文本描述。
  • 适合需要精确控制动作但保持多样外观的视频生成任务。

动作定制旨在使扩散模型(DM)能够生成具有指定动作概念的视频,该概念由一组具有相同动作的视频片段定义。为实现此目标,模型在适配过程中需仅建模特定动作概念,而不影响生成多样化外观的能力。因此,关键在于如何在适配过程中将动作与外观分离。以往方法尝试通过学习动作LoRA、使用潜在噪声残差等方式将动作概念注入大规模预训练文本到视频扩散模型中。然而这些方法不可避免地编码了参考视频中的外观信息,削弱了外观生成能力。本文沿用学习动作LoRA的方式,提出两种新策略以增强动作-外观分离:时间注意力净化(TAP)和外观高速通道(AH)。具体而言,假设在时间注意力模块中,预训练的Value嵌入已足够作为生成新动作的基本组件。因此,在TAP中,仅通过动作LoRA重塑时间注意力,使Value嵌入重新组织以产生新动作。此外,在AH中,将U-Net中每个跳跃连接的起始点从时间注意力模块输出改为空间注意力模块输出。大量实验表明,相比先前方法,本方法生成的视频在外观上更贴近文本描述,动作上更一致于参考视频。

原文摘要 · Abstract (English)

Motion customization aims to adapt the diffusion model (DM) to generate videos with the motion specified by a set of video clips with the same motion concept. To realize this goal, the adaptation of DM should be possible to model the specified motion concept, without compromising the ability to generate diverse appearances. Thus, the key to solving this problem lies in how to separate the motion concept from the appearance in the adaptation process of DM. Typical previous works explore different ways to represent and insert a motion concept into large-scale pretrained text-to-video diffusion models, e.g., learning a motion LoRA, using latent noise residuals, etc. While those methods can encode the motion concept, they also inevitably encode the appearance in the reference videos, resulting in weakened appearance generation capability. In this paper, we follow the typical way to learn a motion LoRA to encode the motion concept, but propose two novel strategies to enhance motion-appearance separation, including temporal attention purification (TAP) and appearance highway (AH). Specifically, we assume that in the temporal attention module, the pretrained Value embeddings are sufficient to serve as basic components needed by producing a new motion. Thus, in TAP, we choose only to reshape the temporal attention with motion LoRAs so that Value embeddings can be reorganized to produce a new motion. Further, in AH, we alter the starting point of each skip connection in U-Net from the output of each temporal attention module to the output of each spatial attention module. Extensive experiments demonstrate that compared to previous works, our method can generate videos with appearance more aligned with the text descriptions and motion more consistent with the reference videos.

视频生成扩散模型动作控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。