arXiv:2409.03514cs.CV2024-09被引 3

用注意力控制改进扩散模型,实现视频局部编辑的背景保留与动作连贯。

Blended Latent Diffusion under Attention Control for Real-World Video Editing

  • 用DDIM反演获取背景潜在表示,替代随机噪声,提升背景保真度。
  • 通过交叉注意力图自动生成编辑掩码,无需用户手动标注。
  • 将U-Net自注意力块改为时空块,增强视频帧间运动一致性。

由于缺乏公开可用的文本到视频生成模型,现有视频编辑方法多基于预训练的文本到图像生成模型,但在处理含时序信息的视频局部编辑时仍面临挑战。首先,现有方法虽通过预定义掩码实现局部编辑,但因每帧整体生成,难以保持外部区域背景;其次,用户手动提供掩码成本高,亟需集成于编辑流程中的自动掩码策略;最后,图像级预训练模型未学习跨帧时序信息,而该信息对表达运动与动态至关重要。本文提出将图像级混合潜空间扩散模型适配用于局部视频编辑任务。具体而言,利用DDIM反演获取背景潜在变量,而非随机噪声,以更好保留输入视频的背景信息;进一步引入基于扩散步骤中交叉注意力图的自主掩码生成机制;最后,通过将U-Net的自注意力块转换为时空块,增强视频帧间的时序一致性。大量实验表明,所提方法在多种真实场景视频编辑任务中均表现出色。

原文摘要 · Abstract (English)

Due to lack of fully publicly available text-to-video models, current video editing methods tend to build on pre-trained text-to-image generation models, however, they still face grand challenges in dealing with the local editing of video with temporal information. First, although existing methods attempt to focus on local area editing by a pre-defined mask, the preservation of the outside-area background is non-ideal due to the spatially entire generation of each frame. In addition, specially providing a mask by user is an additional costly undertaking, so an autonomous masking strategy integrated into the editing process is desirable. Last but not least, image-level pretrained model hasn't learned temporal information across frames of a video which is vital for expressing the motion and dynamics. In this paper, we propose to adapt a image-level blended latent diffusion model to perform local video editing tasks. Specifically, we leverage DDIM inversion to acquire the latents as background latents instead of the randomly noised ones to better preserve the background information of the input video. We further introduce an autonomous mask manufacture mechanism derived from cross-attention maps in diffusion steps. Finally, we enhance the temporal consistency across video frames by transforming the self-attention blocks of U-Net into temporal-spatial blocks. Through extensive experiments, our proposed approach demonstrates effectiveness in different real-world video editing tasks.

视频编辑扩散模型时序一致

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。