提出自梯度强迫机制,让模型学会更好写入记忆以生成更连贯长视频。
Self Gradient Forcing: Native Long Video Extrapolation

- 两阶段训练:先自生成历史,再用未来帧反向优化记忆写入方式
- 仅用5秒训练数据即可生成数分钟长视频,显著提升连贯性与稳定性
- 适合研究长视频生成、扩散模型和自回归建模的开发者与研究者
近期自回归视频扩散方法普遍采用自强制(Self Forcing),即学生模型在自身生成的历史上训练,而非真实视频上下文,从而降低暴露偏差。然而,历史键值缓存仍作为固定状态被后续帧使用,导致未来损失无法监督早期生成潜在表示如何被写入更有效的键值以服务后续生成。我们称此为历史上下文-梯度缺口。为此提出自梯度强迫(SGF),一种两阶段训练策略,在不反向传播完整序列的前提下恢复缺失的监督信号。第一阶段进行无梯度自回归滚动推断,在采样的去噪退出步骤记录自生成上下文及输入模型的噪声潜在表示;第二阶段并行重构该步骤的上下文梯度,将生成上下文作为停梯度的清洁潜在输入,重新计算上下文键值表示与未来到上下文的因果注意力。因此,SGF在原生自回归训练目标内提供了缺失的记忆写入监督,利用未来视频潜在表示的损失训练模型将上下文编码为更有效的因果记忆。在不同初始化下广泛开展的长时序帧级与块级实验表明,相较于自强制,SGF实现了更强的原生长视频外推能力,尤其在主体身份、背景/布局一致性与时间稳定性方面表现优异。值得注意的是,仅使用5秒训练窗口,SGF即可外推至持续数分钟的视频。代码与模型将公开,以推动自回归视频生成研究。
原文摘要 · Abstract (English)
Recent autoregressive video diffusion methods are increasingly built upon Self Forcing, where the student is trained on histories produced by its own rollout rather than ground-truth video contexts. This reduces exposure bias, but the historical key-value cache is still used by future frames only as frozen rollout state. As a result, future losses cannot supervise how earlier generated latents should be written into more useful keys and values for later video-latent generation. We call this the historical context-gradient gap. We propose Self Gradient Forcing (SGF), a two-pass training strategy that restores this missing supervision signal without backpropagating through the full serial rollout. Pass 1 performs a no-gradient autoregressive rollout matching inference and, at a sampled denoising exit step, records both the self-generated context and the noisy latents fed to the model. Pass 2 performs parallel context-gradient reconstruction for the recorded exit step. The generated context is used as stop-gradient clean-latent input, while the model recomputes the context KV representations and future-to-context causal attention. Thus, SGF provides the missing memory-writing supervision within the native autoregressive training objective, using losses on future video latents to train the model to encode context into more effective causal memory. Across extensive long-horizon frame-wise and chunk-wise experiments under different initializations, SGF achieves stronger native long-video extrapolation than Self Forcing, especially in subject identity, background/layout consistency, and temporal stability. Remarkably, using only a 5-second training window, SGF can extrapolate to videos lasting several minutes. Code and models will be released to advance research on autoregressive video generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。