通过推理时剔除不稳定的潜在令牌,缓解自回归长视频生成中的时序漂移问题。
TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation
- 在推理阶段识别并删除与前一批次差异显著的异常潜在令牌。
- 在长视频生成中显著提升时序一致性,未修改模型结构或训练流程。
- 适用于需高时序稳定性的自回归视频生成任务,如长视频创作。
自回归视频生成通过逐批条件生成实现长视频合成,但近期研究发现此类流程存在严重时序漂移,错误随时间累积并放大。我们假设该漂移并非源于模型容量不足,而是推理过程中的误差传播所致。具体而言,漂移源于自回归推理中对已损坏潜在条件令牌的不受控重复使用。为此,我们提出一种简单、基于推理阶段的方法:通过识别并移除即将被重复使用的不稳定潜在令牌,以缓解误差积累。我们将不稳定令牌定义为表示与先前生成批次显著偏离的潜在令牌,表明可能存在损坏或语义漂移。通过显式从自回归上下文中移除这些受损潜在令牌,而非修改整个空间区域或模型参数,该方法防止不可靠的潜在信息影响后续生成步骤。结果表明,该方法在不修改模型架构、训练过程或潜在空间的情况下,显著提升了长时域的时序一致性。
原文摘要 · Abstract (English)
Auto-regressive video generation enables long video synthesis by iteratively conditioning each new batch of frames on previously generated content. However, recent work has shown that such pipelines suffer from severe temporal drift, where errors accumulate and amplify over long horizons. We hypothesize that this drift does not primarily stem from insufficient model capacity, but rather from inference-time error propagation. Specifically, we contend that drift arises from the uncontrolled reuse of corrupted latent conditioning tokens during auto-regressive inference. To correct this accumulation of errors, we propose a simple, inference-time method that mitigates temporal drift by identifying and removing unstable latent tokens before they are reused for conditioning. For this purpose, we define unstable tokens as latent tokens whose representations deviate significantly from those of the previously generated batch, indicating potential corruption or semantic drift. By explicitly removing corrupted latent tokens from the auto-regressive context, rather than modifying entire spatial regions or model parameters, our method prevents unreliable latent information from influencing future generation steps. As a result, it significantly improves long-horizon temporal consistency without modifying the model architecture, training procedure, or leaving latent space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。