通过调控时空注意力，实现视频的多粒度精细编辑。
VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing
- 用注意力机制分离空间区域，精准响应局部提示。
- 在真实场景中达到当前最佳编辑效果。
- 适合需要细粒度控制的视频生成研究者。
扩散模型的进展显著提升了视频生成与编辑能力，但涵盖类别级、实例级和部件级修改的多粒度视频编辑仍是重大挑战。主要困难在于文本到区域控制的语义错位以及扩散模型中的特征耦合。为此，我们提出 VideoGrain，一种零样本方法，通过调节时空（交叉与自）注意力机制，实现对视频内容的细粒度控制。通过增强每个局部提示对其对应空间解耦区域的注意力，同时抑制其与无关区域的交叉注意力交互，提升文本到区域的控制精度。此外，通过增强区域内感知并减少区域间干扰，改善特征分离。大量实验证明，该方法在真实场景中达到业界领先性能。代码、数据与演示已公开于 https://knightyxp.github.io/VideoGrain_project_page/
原文摘要 · Abstract (English)
Recent advancements in diffusion models have significantly improved video generation and editing capabilities. However, multi-grained video editing, which encompasses class-level, instance-level, and part-level modifications, remains a formidable challenge. The major difficulties in multi-grained editing include semantic misalignment of text-to-region control and feature coupling within the diffusion model. To address these difficulties, we present VideoGrain, a zero-shot approach that modulates space-time (cross- and self-) attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt's attention to its corresponding spatial-disentangled region while minimizing interactions with irrelevant areas in cross-attention. Additionally, we improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention. Extensive experiments demonstrate our method achieves state-of-the-art performance in real-world scenarios. Our code, data, and demos are available at https://knightyxp.github.io/VideoGrain_project_page/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。