解决自回归视频生成模型长时滚动中的质量退化问题
Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion
- 提出Rolling Sink缓存策略，动态管理生成过程中的历史内容
- 在5秒训练基础上实现30分钟稳定生成，质量显著优于基线
- 无需重新训练，适合长期视频生成任务的部署与应用
自回归视频扩散模型（如Self Forcing）以块为单位逐块生成视频，可无限滚动。但其在5秒固定片段上训练，一旦超过训练时长，生成缓存中累积的内容超出训练分布，导致色彩过饱和、身份漂移、结构坍塌和运动衰减。本文将此现象归因于缓存行为的领域偏移，并提出Rolling Sink——一种无需重训练的推理时缓存策略。该方法在固定缓存预算下，通过滑动时间索引与语义调度，维持与训练期内缓存相似的稳定状态，有效抑制内容漂移。在两个5秒训练的基线模型（Self Forcing, Causal Forcing）上，Rolling Sink在相同推理设置下，于VBench-Long数据集上1分钟与5分钟滚动生成任务中均取得最佳平均排名，吞吐率仅下降约0.56%，显存占用恒定。可支持长达30分钟的定性生成，达训练时长的360倍。验证覆盖1分钟与5分钟滚动生成，不保证任意时长下的持续质量。
原文摘要 · Abstract (English)
Autoregressive (AR) video diffusion models in the Self Forcing family generate a video chunk by chunk, each conditioned on the ones before it, so they can keep rolling out with no preset length. In practice they are trained on short, fixed clips (e.g., 5s). Once a rollout passes this training horizon, the cache fills with self-generated chunks the model never saw in training, and quality degrades: colors oversaturate, identities drift, structures collapse, and motion fades. We frame this failure as a cache-behavior mismatch, a domain shift between the within-horizon caches seen in training and those produced by open-ended testing, and address it with Rolling Sink, a training-free inference-time cache policy. Within the base model's fixed cache budget, Rolling Sink holds the cache close to its stable within-horizon behavior: it restores low-drift content, sliding temporal indices, and evolving semantics, and keeps a small recent context. Attention sink and temporal re-indexing are existing mechanisms. Our contribution is the mismatch formulation, the joint bounded-cache policy, and the Rolling Semantics schedule. On two 5s-trained bases, Self Forcing and Causal Forcing, Rolling Sink attains the best averaged rank on VBench-Long at both 1- and 5-minute rollouts under the same inference setting, at about 0.56% throughput change, and holds GPU memory constant. It also sustains a 30-minute qualitative rollout, 360 times its training horizon. We validate at 1- and 5-minute rollouts and do not claim guaranteed quality at arbitrary duration. Project page: https://rolling-sink.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。