实现低延迟视频编辑,保持背景稳定且实时响应。
LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing

- 分三阶段蒸馏技术,将强大模型能力迁移到轻量流式编辑器。
- 实测推理速度达12.66 FPS,显著提升交互效率。
- 专为实时编辑设计,适合AR与互动应用场景。
流式视频编辑进展迅速,但实际部署仍受限于两个核心问题:长时间保持背景和未编辑区域的稳定性,以及满足实时交互所需的低延迟。当前主流流式视频生成方法多用于内容合成,无法直接用于编辑,因编辑需严格保留原内容并实现区域精准控制。本文提出一种新型流式视频编辑框架,支持因果性、逐帧编辑,在保持强内容一致性的同时实现实时响应。核心设计为三阶段知识蒸馏流程,将强大的双向基础模型编辑能力逐步迁移至高效单向流式编辑器,实现长时程稳定编辑且不损失视觉质量。为进一步支持实时部署,引入面向AR的掩码缓存机制,复用跨帧的区域计算,大幅减少冗余处理,加速推理过程。最后,我们建立了专门的流式视频编辑基准。大量实验表明,本方法在流式基线中达到最先进视觉质量,同时推理速度提升至12.66 FPS,适用于交互式与增强现实应用场景。
原文摘要 · Abstract (English)
Streaming video editing has made rapid progress, yet practical deployment is still limited by two core issues: maintaining stable backgrounds and non-edited regions over time, and achieving the low latency required for real-time interactive scenarios. Meanwhile, recent streaming video generation methods are mostly developed for synthesis and cannot be directly applied to editing due to the strict preservation requirement and region-specific control. In this work, we present a novel streaming video editing framework that performs causal, frame-by-frame editing with strong content preservation and real-time responsiveness. Our key design is a three-stage distillation pipeline that progressively transfers editing capability from a powerful bidirectional foundation model to an efficient unidirectional streaming editor, enabling stable long-horizon edits without sacrificing visual fidelity. To further support real-time deployment, we introduce an AR-oriented mask cache that reuses region-related computation across frames, substantially reducing redundant processing and accelerating inference. Finally, we establish a dedicated benchmark for streaming video editing. Extensive evaluations demonstrate that our method achieves state-of-the-art visual quality among streaming baselines while drastically boosting inference speed to 12.66 FPS, making it suitable for interactive and augmented reality applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。