通过自适应剪枝与关键帧选择，实现分钟级长视频高效编辑
AdaFlow: Efficient Long Video Editing via Adaptive Attention Slimming And Keyframe Selection
- 基于注意力重要性动态压缩键帧序列，提升处理长度
- 单次推理生成超1000帧视频，是对比方法的10倍
- 无需训练，适合长视频生成与编辑任务
尽管取得进展，文本驱动的长视频编辑仍因内存开销过大而困难重重。现有方法虽将任务简化为键帧转换与插值生成两步，但逐标记的键帧转换仍限制了视频长度上限。本文提出一种新颖且无需训练的高效长视频编辑方法AdaFlow。我们发现视频帧中并非所有标记对键帧转换同等重要，据此提出自适应注意力剪枝机制，压缩$KV$序列，使可处理的键帧数量提升一个数量级。同时引入自适应关键帧选择机制，选取代表性帧进行联合编辑，进一步提升生成质量。得益于这些设计，AdaFlow可在单次推理中完成分钟级高质量长视频编辑，即在单块A800 GPU上处理超过1000帧，比对比方法（如TokenFlow）长十倍。为验证效果，我们还构建了新的高精度标注基准LongV-EVAL。代码已开源。
原文摘要 · Abstract (English)
Despite great progress, text-driven long video editing is still notoriously challenging mainly due to excessive memory overhead. Although recent efforts have simplified this task into a two-step process of keyframe translation and interpolation generation, the token-wise keyframe translation still plagues the upper limit of video length. In this paper, we propose a novel and training-free approach towards efficient and effective long video editing, termed AdaFlow. We first reveal that not all tokens of video frames hold equal importance for keyframe translation, based on which we propose an Adaptive Attention Slimming scheme for AdaFlow to squeeze the $KV$ sequence, thus increasing the number of keyframes for translations by an order of magnitude. In addition, an Adaptive Keyframe Selection scheme is also equipped to select the representative frames for joint editing, further improving generation quality. With these innovative designs, AdaFlow achieves high-quality long video editing of minutes in one inference, i.e., more than 1$k$ frames on one A800 GPU, which is about ten times longer than the compared methods, e.g., TokenFlow. To validate AdaFlow, we also build a new benchmark for long video editing with high-quality annotations, termed LongV-EVAL. Our code is released at: https://github.com/jidantang55/AdaFlow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。