让视频预测模型自动调节噪声，持续适应连续视频流。
Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimization
- 冻结模型参数，仅优化推理时的扩散噪声以自适应调整。
- 在Ego4D等数据集上，FVD、SSIM、PSNR均显著提升。
- 适合需要长期在线更新的视频预测场景。
本文研究基于扩散模型的视频预测方法，用于连续视频流的未来帧生成。在此场景下，模型持续接收新样本，我们旨在利用这一特性提升预测效果。为此，提出一种持续适应预训练扩散模型的方法：不微调模型参数，而是在推理时优化扩散噪声，使模型能自适应选择合适采样噪声。该方法称为序列自适应视频预测与扩散噪声优化（SAVi-DNO）。为验证其有效性，我们在Ego4D数据集上设计了新的评估设置，聚焦于长时连续视频的同步适应与评估。实验表明，在Ego4D、OpenDV-YouTube、UCF-101和SkyTimelapse数据集上，基于FVD、SSIM和PSNR指标，模型性能均有明显提升，证明了SAVi-DNO的有效性。
原文摘要 · Abstract (English)
In this work, we investigate diffusion-based video prediction models, which forecast future video frames, for continuous video streams. In this context, the models observe continuously new training samples, and we aim to leverage this to improve their predictions. We thus propose an approach that continuously adapts a pre-trained diffusion model to a video stream. Since fine-tuning the parameters of a large diffusion model is too expensive, we refine the diffusion noise during inference while keeping the model parameters frozen, allowing the model to adaptively determine suitable sampling noise. We term the approach Sequence Adaptive Video Prediction with Diffusion Noise Optimization (SAVi-DNO). To validate our approach, we introduce a new evaluation setting on the Ego4D dataset, focusing on simultaneous adaptation and evaluation on long continuous videos. Empirical results demonstrate improved performance based on FVD, SSIM, and PSNR metrics on long videos of Ego4D and OpenDV-YouTube, as well as videos of UCF-101 and SkyTimelapse, showcasing SAVi-DNO's effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。