用蒸馏技术让视频修复快到实时,质量不降。
InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior
- 把复杂视频扩散模型蒸馏成单次前传的快速模型,无需迭代优化。
- 在多种视频修复任务中达到顶尖质量,速度超35帧/秒。
- 适合需要低延迟的直播、AR/VR等实时交互场景。
视频逆问题对流媒体、远程通信和虚拟现实至关重要,需在低延迟下保持高感知质量。当前基于扩散模型的方法要么用图像扩散模型加临时正则化,导致时间伪影;要么依赖原生视频扩散模型,其迭代采样过慢,无法实时使用。我们提出 InstantViR,一种基于预训练视频扩散先验的快速推理框架。通过将强大的双向视频扩散模型(教师)蒸馏为因果自回归学生模型,实现单次前向传播即可完成退化视频到恢复视频的映射,继承教师的强时序建模能力,同时完全消除测试时的迭代优化。蒸馏过程仅需教师扩散模型和已知退化算子,无需外部成对的干净/噪声视频数据。为进一步提升吞吐量,采用创新的教师空间正则化蒸馏方案,将视频扩散骨干 VAE 替换为高效轻量级 LeanVAE,实现低延迟潜空间处理。在流媒体随机补全、高斯模糊去噪和超分辨率任务中,InstantViR 在重建质量上媲美或超越基于扩散的基线方法,且在 NVIDIA A100 GPU 上运行速度超过 35 FPS,较迭代视频扩散求解器最高提速 100 倍。结果表明,基于扩散的视频重建可满足实时、交互式、可编辑的流媒体需求,使其成为现代视觉系统中的实用组件。
原文摘要 · Abstract (English)
Video inverse problems are fundamental to streaming, telepresence, and AR/VR, where high perceptual quality must coexist with tight latency constraints. Diffusion-based priors currently deliver state-of-the-art reconstructions, but existing approaches either adapt image diffusion models with ad hoc temporal regularizers - leading to temporal artifacts - or rely on native video diffusion models whose iterative posterior sampling is far too slow for real-time use. We introduce InstantViR, an amortized inference framework for ultra-fast video reconstruction powered by a pre-trained video diffusion prior. We distill a powerful bidirectional video diffusion model (teacher) into a causal autoregressive student that maps a degraded video directly to its restored version in a single forward pass, inheriting the teacher's strong temporal modeling while completely removing iterative test-time optimization. The distillation is prior-driven: it only requires the teacher diffusion model and known degradation operators, and does not rely on externally paired clean/noisy video data. To further boost throughput, we replace the video-diffusion backbone VAE with a high-efficiency LeanVAE via an innovative teacher-space regularized distillation scheme, enabling low-latency latent-space processing. Across streaming random inpainting, Gaussian deblurring and super-resolution, InstantViR matches or surpasses the reconstruction quality of diffusion-based baselines while running at over 35 FPS on NVIDIA A100 GPUs, achieving up to 100 times speedups over iterative video diffusion solvers. These results show that diffusion-based video reconstruction is compatible with real-time, interactive, editable, streaming scenarios, turning high-quality video restoration into a practical component of modern vision systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。