首个在消费级显卡实现实时1080p视频修复的生成模型
SwiftVR: Real-Time One-Step Generative Video Restoration

- 用无掩码移位窗口自注意力减少高分辨率计算开销
- 轻量修复感知编码器实现快速逐块解码,3840x2160下达14帧/秒
- 无需重训练或专用内核,可直接部署在普通显卡上
实时直播视频修复要求在严格单帧延迟约束下输出高分辨率画面。现有一步扩散模型因两个主要瓶颈难以在消费级显卡上部署:高分辨率下的二次方空间注意力和大型视频自编码器带来的延迟-内存开销。我们提出SwiftVR,一种基于因果分块协议的实时一步生成式视频修复框架,同时缓解这两类瓶颈。注意力方面,无掩码移位窗口自注意力通过确定性索引将每个空间窗口聚合为密集张量,全程使用标准密集缩放点积注意力,避免掩码、循环移位、填充或硬件特定稀疏内核。由于仅使用标准密集SDPA调用,训练好的模型可直接迁移至消费级显卡,无需重训练或定制内核。自编码方面,轻量级修复感知自编码器支持快速分块解码,同时保持重建质量。在单个H100上,SwiftVR在2560x1440下维持31~41帧/秒,在3840x2160下达14帧/秒,而所有对比的扩散模型在4K下均超出内存限制。在消费级RTX 5090上,可达1920x1080下26~31帧/秒。据我们所知,SwiftVR是首个在消费级显卡上实现实时1080p流媒体生成视频修复的模型,且以更低推理成本达到强无参考感知质量。
原文摘要 · Abstract (English)
Real-time video restoration (VR) for live streams requires high-resolution outputs under strict per-frame latency constraints. Existing one-step diffusion-based VR models remain difficult to deploy on consumer-grade GPUs due to two main bottlenecks: quadratic spatial attention at high resolutions and the latency-memory overhead of large video autoencoders. We present SwiftVR, a streaming one-step generative VR framework that reduces both bottlenecks under a causal chunk-wise protocol. For attention, mask-free shifted-window self-attention gathers each spatial window into a dense tensor via deterministic indexing, keeping all attention calls on the dense scaled dot-product attention path without masks, cyclic shifts, padding, or hardware-specific sparse kernels. Because SwiftVR uses only standard dense SDPA calls, the trained model transfers to consumer GPUs without retraining or custom kernels. For autoencoding, a lightweight Restoration-aware Autoencoder enables fast chunk-wise decoding while preserving reconstruction quality. On a single H100, SwiftVR sustains 31~FPS at 2560x1440 and 14~FPS at 3840x2160, whereas all compared diffusion-based VR baselines exceed the memory limit at 4K. On a consumer RTX~5090, SwiftVR reaches 26~FPS at 1920x1080. To our knowledge, SwiftVR is the first generative VR model to achieve real-time 1080p streaming on a consumer-grade GPU, while attaining strong no-reference perceptual quality with lower inference cost. Project is available at https://h-oliday.github.io/SwiftVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。