arXiv:2512.21015cs.CV2025-12中稿 · IEEE Transactions …被引 3

用Mamba替代注意力机制,让视频编辑更快更稳

FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing

  • 用线性时序模块Mamba替代传统注意力,提升效率
  • 在真实视频上成功编辑属性、主体和场景,保持一致性
  • 适合需要快速高效视频编辑的研究与应用

大规模文本到图像扩散模型在图像生成与编辑中取得了前所未有的成功。然而,将这一成果拓展到视频编辑仍面临挑战。现有方法通过引入时间注意力机制来适配视频任务,但依然存在时间不一致和计算开销大的问题。本文提出FluencyVE,一种简单高效的单次视频编辑方法。该方法将线性时序模块Mamba集成到基于预训练Stable Diffusion的视频编辑模型中,替代原有的时间注意力层,实现全局帧级注意力的同时降低计算成本。此外,采用低秩近似矩阵替换因果注意力中的查询与键权重矩阵,并在训练中使用加权平均策略更新注意力分数。该方法在保持文本到图像模型生成能力的同时,显著减轻计算负担。实验与分析表明,在真实视频中对多种属性、主体和位置的编辑均取得良好效果。

原文摘要 · Abstract (English)

Large-scale text-to-image diffusion models have achieved unprecedented success in image generation and editing. However, extending this success to video editing remains challenging. Recent video editing efforts have adapted pretrained text-to-image models by adding temporal attention mechanisms to handle video tasks. Unfortunately, these methods continue to suffer from temporal inconsistency issues and high computational overheads. In this study, we propose FluencyVE, which is a simple yet effective one-shot video editing approach. FluencyVE integrates the linear time-series module, Mamba, into a video editing model based on pretrained Stable Diffusion models, replacing the temporal attention layer. This enables global frame-level attention while reducing the computational costs. In addition, we employ low-rank approximation matrices to replace the query and key weight matrices in the causal attention, and use a weighted averaging technique during training to update the attention scores. This approach significantly preserves the generative power of the text-to-image model while effectively reducing the computational burden. Experiments and analyses demonstrate promising results in editing various attributes, subjects, and locations in real-world videos.

视频编辑Mamba扩散模型高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。