arXiv:2412.19489cs.CV2024-12被引 2

用一张显卡实现实时无限长视频动画,画质稳定不卡顿。

RAIN: Real-time Animation of Infinite Video Stream

  • 通过优化跨噪声层级的帧令牌注意力机制,实现高效长视频生成。
  • 在RTX 4090上实现实时生成,延迟更低,视频连续性显著提升。
  • 仅增加少量模块,适合追求高质量实时视频生成的研究与开发者。

实时动画在提升在线互动方面广受欢迎,但使用扩散模型实现高质量、低延迟、稳定的实时动画仍具挑战,尤其在消费级显卡上。现有方法在生成长视频流时效率低下,常受延迟和画质下降影响。本文提出RAIN,一种可在单张RTX 4090 GPU上实现实时无限长视频流动画的流水线方案。其核心在于高效计算不同噪声水平下长时序间隔的帧令牌注意力,同时对远超以往流式方法数量的帧令牌进行联合去噪。该设计使帧生成延迟更短、速度更快,并保持长程依赖关系,从而显著提升视频连续性与一致性。经数个训练周期微调后,基于RAIN的Stable Diffusion模型可实现无限制长度的实时低延迟视频生成,画质与连贯性优于现有方法。尽管功能强大,RAIN仅引入少量1D注意力模块,开销极小。实验在基准数据集及超长视频生成中均验证其优势,展现更高画质、精度与稳定性。所有代码与模型将公开。

原文摘要 · Abstract (English)

Live animation has gained immense popularity for enhancing online engagement, yet achieving high-quality, real-time, and stable animation with diffusion models remains challenging, especially on consumer-grade GPUs. Existing methods struggle with generating long, consistent video streams efficiently, often being limited by latency issues and degraded visual quality over extended periods. In this paper, we introduce RAIN, a pipeline solution capable of animating infinite video streams in real-time with low latency using a single RTX 4090 GPU. The core idea of RAIN is to efficiently compute frame-token attention across different noise levels and long time-intervals while simultaneously denoising a significantly larger number of frame-tokens than previous stream-based methods. This design allows RAIN to generate video frames with much shorter latency and faster speed, while maintaining long-range attention over extended video streams, resulting in enhanced continuity and consistency. Consequently, a Stable Diffusion model fine-tuned with RAIN in just a few epochs can produce video streams in real-time and low latency without much compromise in quality or consistency, up to infinite long. Despite its advanced capabilities, the RAIN only introduces a few additional 1D attention blocks, imposing minimal additional burden. Experiments in benchmark datasets and generating super-long videos demonstrating that RAIN can animate characters in real-time with much better quality, accuracy, and consistency than competitors while costing less latency. All code and models will be made publicly available.

视频生成扩散模型实时动画长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。