用线性复杂度模型实现高效视频编辑,兼顾速度与质量。
VRWKV-Editor: Reducing quadratic complexity in transformer-based video editing
- 引入线性时空聚合模块,替代传统注意力机制
- 相比顶尖方法提速3.7倍,内存降低60%
- 适合长时高清视频的实时编辑场景
随着视频编辑技术的发展,聚焦时空依赖的深度学习模型已成为主流。但这类模型受限于传统注意力机制带来的二次方计算复杂度,难以处理长时长、高分辨率视频,限制了其在实时视频处理等实际场景的应用。为此,本文提出VRWKV-Editor,一种将线性时空聚合模块融入基于扩散模型的视频编辑框架的新方法。该模型利用RWKV Transformer的双向加权键值递归机制,捕捉全局依赖并保持时间连贯性,实现线性复杂度而不损失质量。大量实验表明,该方法相较当前最优扩散模型实现最高3.7倍加速和60%内存降低,同时在帧一致性与文本对齐性能上保持竞争力。针对不同序列长度的对比分析进一步验证:随着视频变长,本方法与自注意力架构间的编辑速度差距显著扩大。
原文摘要 · Abstract (English)
In light of recent progress in video editing, deep learning models focusing on both spatial and temporal dependencies have emerged as the primary method. However, these models suffer from the quadratic computational complexity of traditional attention mechanisms, making them difficult to adapt to long-duration and high-resolution videos. This limitation restricts their applicability in practical contexts such as real-time video processing. To tackle this challenge, we introduce a method to reduce both time and space complexity of these systems by proposing VRWKV-Editor, a novel video editing model that integrates a linear spatio-temporal aggregation module into video-based diffusion models. VRWKV-Editor leverages bidirectional weighted key-value recurrence mechanism of the RWKV transformer to capture global dependencies while preserving temporal coherence, achieving linear complexity without sacrificing quality. Extensive experiments demonstrate that the proposed method achieves up to 3.7x speedup and 60% lower memory usage compared to state-of-the-art diffusion-based video editing methods, while maintaining competitive performance in frame consistency and text alignment. Furthermore, a comparative analysis we conducted on videos with different sequence lengths confirms that the gap in editing speed between our approach and architectures with self-attention becomes more significant with long videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。