提出高效长视频生成方法,解决训练成本高与长期依赖难题
Long-Context Autoregressive Video Modeling with Next-Frame Prediction
- 用自回归框架建模连续帧,区分局部细节与远程上下文
- 通过非对称分块核降低冗余特征,训练成本显著下降
- 在短/长视频生成上均达领先效果,适合视频世界模型研究
长时序视频建模对于生成模型作为世界模拟器至关重要,需在长时间跨度中保持时间一致性。然而多数现有模型仅在短片段上训练,难以捕捉长程依赖关系,即使测试时外推也受限。直接在长视频上训练虽是自然方案,但视觉标记数量快速增长导致计算开销巨大。为此,我们建立一个强基准模型——帧自回归(FAR),其建模连续帧间的时间依赖,收敛速度优于视频扩散变换器,且超越词级别自回归模型。基于此,我们发现视频自回归中存在上下文冗余:邻近帧对时间一致性至关重要,而远距离帧主要起记忆作用。为消除冗余,提出使用非对称分块核的长短程上下文建模策略:对远距离帧采用大核减少冗余标记,对局部帧使用标准核保留细粒度信息。该方法显著降低长视频训练成本。所提方法在短视频与长视频生成任务上均达到当前最优性能,为长时序自回归视频建模提供了有效基线。
原文摘要 · Abstract (English)
Long-context video modeling is essential for enabling generative models to function as world simulators, as they must maintain temporal coherence over extended time spans. However, most existing models are trained on short clips, limiting their ability to capture long-range dependencies, even with test-time extrapolation. While training directly on long videos is a natural solution, the rapid growth of vision tokens makes it computationally prohibitive. To support exploring efficient long-context video modeling, we first establish a strong autoregressive baseline called Frame AutoRegressive (FAR). FAR models temporal dependencies between continuous frames, converges faster than video diffusion transformers, and outperforms token-level autoregressive models. Based on this baseline, we observe context redundancy in video autoregression. Nearby frames are critical for maintaining temporal consistency, whereas distant frames primarily serve as context memory. To eliminate this redundancy, we propose the long short-term context modeling using asymmetric patchify kernels, which apply large kernels to distant frames to reduce redundant tokens, and standard kernels to local frames to preserve fine-grained detail. This significantly reduces the training cost of long videos. Our method achieves state-of-the-art results on both short and long video generation, providing an effective baseline for long-context autoregressive video modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。