用自回归扩散模型加速视频逆问题求解，实现近实时恢复。
Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models
- 采用自回归架构逐帧流式重建视频，消除初始延迟瓶颈。
- 初始延迟从114秒降至4秒，吞吐量提升至5.91 FPS。
- 适合需要低延迟、高效率视频修复的应用场景。
扩散模型为零样本视频逆问题提供了强大先验，但其实时部署受限于两个效率问题：整体视频恢复导致的高初始延迟，以及为在像素空间保持测量一致性而需多次VAE推理造成的低吞吐量。为此，我们提出自回归视频逆问题求解器（AVIS）。AVIS框架利用自回归视频扩散模型以流式方式重建视频，天然消除了延迟瓶颈。具体而言，AVIS通过测量一致的估计初始化反向扩散过程，减少了所需采样步数。相比领先的非自回归求解器，AVIS将初始延迟从114秒降至4秒，吞吐量从0.71提升至1.18 FPS，同时实现更优的恢复质量。我们进一步提出加速版本AVIS Flash，仅在首个视频块上强制测量一致性。AVIS Flash在单张RTX 4090 GPU上将吞吐量大幅提升至5.91 FPS，保持竞争力性能，达成理想的效率-性能权衡，为实时部署铺平道路。
原文摘要 · Abstract (English)
Diffusion models provide powerful priors for zero-shot video inverse problems, but their real-time deployment is hindered by two inefficiencies: high initial latency caused by holistic video restoration, and low throughput resulting from multiple VAE passes to enforce measurement consistency in pixel space. To overcome these limitations, we propose Autoregressive Video Inverse problem Solver (AVIS). The AVIS framework leverages autoregressive video diffusion models to restore videos in a streaming manner, naturally eliminating latency bottlenecks. Specifically, AVIS initializes reverse diffusion with a measurement-consistent estimate, reducing the required sampling steps. Compared to leading non-autoregressive solvers, AVIS drastically reduces initial latency from 114s to 4s and increases throughput from 0.71 to 1.18 FPS while achieving superior restoration quality. We further introduce a highly accelerated variant, dubbed AVIS Flash, that enforces measurement consistency solely on the first chunk. AVIS Flash substantially boosts throughput to 5.91 FPS on a single RTX 4090 GPU while maintaining competitive performance and achieving a favorable efficiency-performance trade-off, paving the way toward real-time deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。