用Transformer实现低延迟视频生成，解码速度提升12倍且内存减少11倍。
FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformers
- 采用滚动键值缓存机制，固定时间窗口内只关注历史帧，保持低延迟。
- 在1080p下重建质量媲美卷积解码器（PSNR 41.55dB），但速度快3.6-12倍。
- 适合实时视频生成场景，尤其适用于高分辨率、长视频流式处理。
实时视频生成不仅需要快速去噪，还需快速解码。现有基于3D卷积的潜空间视频扩散模型在高分辨率或长视频下解码慢且内存占用高。我们提出FlashDecoder，一种纯Transformer架构的快速、低内存视频解码器，逐帧将潜变量解码为像素。每步仅通过滚动键值缓存关注固定大小的历史帧窗口，确保解码速度恒定、内存有限，支持常数延迟流式传输。因帧按顺序处理，自然满足时序因果性，无需显式注意力掩码，可训练至1080p分辨率，重建质量与卷积解码器相当。在Wan2.1和Wan2.2潜空间上，其重建质量与卷积解码器持平（如1080p下PSNR为41.55dB vs. 41.49dB），在单张H100 GPU上解码速度提升3.6-4.7倍，内存降低最多11倍；经架构感知推理优化后，速度提升达12倍。
原文摘要 · Abstract (English)
Real-time video generation demands fast decoding as much as fast denoising, yet current latent video diffusion models rely on 3D convolutional decoders that are slow and memory-intensive at high resolutions or for long video. We introduce FlashDecoder, a fast, memory-efficient pure-Transformer video decoder that decodes latents to pixels frame by frame. At each step, the current frame attends only to a fixed-size window of past frames through a rolling KV cache. The fixed temporal window keeps decoding fast and memory bounded regardless of video length, enabling constant-latency streaming. Because frames are processed sequentially, temporal causality is enforced without explicit attention masks, enabling training at resolutions up to 1080p and matching the reconstruction quality of convolutional decoders. On the Wan2.1 and Wan2.2 latent spaces, FlashDecoder matches each convolutional decoder in reconstruction quality (e.g., 41.55dB vs. 41.49dB PSNR at 1080p) while decoding 3.6x-4.7x faster with up to 11x less memory on a single H100 GPU. With architecture-aware inference optimizations, the speedup widens to 12x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。