用自回归扩散模型全局建模视频,提升插帧的时序一致性。
Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers
- 采用自回归扩散变换器全局建模整段视频,避免局部处理导致的不一致。
- 在多个基准上达到顶尖性能,大运动场景下时序稳定性和画质显著提升。
- 支持任意分辨率推理且无需重训练,适合高阶视频修复与生成任务。
现有视频插帧(VFI)方法多采用以帧为中心的策略,将视频切分为独立短片段(如三元组)处理,导致时序不一致和运动伪影。为此,我们提出一种全新的、以视频为中心的全局建模范式——局部扩散强制插帧(LDF-VFI)。该框架基于自回归扩散变换器,对完整视频序列进行建模,确保长程时序连贯性。为缓解自回归生成中的误差累积问题,引入新颖的跳跃拼接采样策略,有效维持时序稳定性。此外,LDF-VFI结合稀疏局部注意力与分块VAE编码,不仅高效处理长序列,还可在推理时无须重训练即可适配任意空间分辨率(如4K)。增强的条件VAE解码器利用输入视频的多尺度特征,进一步提升重建保真度。实验表明,LDF-VFI在挑战性VFI基准上取得当前最优表现,尤其在大运动场景中展现出卓越的逐帧质量与时序一致性。源代码已公开于 https://github.com/xypeng9903/LDF-VFI。
原文摘要 · Abstract (English)
Existing video frame interpolation (VFI) methods often adopt a frame-centric approach, processing videos as independent short segments (e.g., triplets), which leads to temporal inconsistencies and motion artifacts. To overcome this, we propose a holistic, video-centric paradigm named Local Diffusion Forcing for Video Frame Interpolation (LDF-VFI). Our framework is built upon an auto-regressive diffusion transformer that models the entire video sequence to ensure long-range temporal coherence. To mitigate error accumulation inherent in auto-regressive generation, we introduce a novel skip-concatenate sampling strategy that effectively maintains temporal stability. Furthermore, LDF-VFI incorporates sparse, local attention and tiled VAE encoding, a combination that not only enables efficient processing of long sequences but also allows generalization to arbitrary spatial resolutions (e.g., 4K) at inference without retraining. An enhanced conditional VAE decoder, which leverages multi-scale features from the input video, further improves reconstruction fidelity. Empirically, LDF-VFI achieves state-of-the-art performance on challenging VFI benchmarks, demonstrating superior per-frame quality and temporal consistency, especially in scenes with large motion. The source code is available at https://github.com/xypeng9903/LDF-VFI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。