通过局部优化与表征连续性，训练速度翻倍且视频质量不降。
Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity
- 在局部窗口内优化生成令牌，结合上下文减少误差传播。
- 使用表征连续性损失，使视频生成更连贯，误差累积降低。
- 实测训练成本减半，适用于需高效训练的文本到视频模型。
自回归模型在图像生成中表现优异且效率高，但在视频生成中仍受高计算开销和长训练时间限制。本文通过实证分析探索加速自回归视频生成模型训练的方法。结果表明，减少训练帧数虽能显著缩短训练时间，但会加剧误差累积并导致生成视频不一致。为此，提出局部优化（Local Opt.）方法，在局部窗口内优化令牌的同时利用上下文信息以减少误差传播。受Lipschitz连续性启发，提出表征连续性（ReCo）策略，通过连续性损失约束表示变化，提升模型鲁棒性并减少误差累积。在类别和文本到视频数据集上的大量实验表明，该方法在性能优于基线的同时，将训练成本降低一半，且不牺牲生成质量。
原文摘要 · Abstract (English)
Autoregressive models have shown superior performance and efficiency in image generation, but remain constrained by high computational costs and prolonged training times in video generation. In this study, we explore methods to accelerate training for autoregressive video generation models through empirical analyses. Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos. To address these issues, we propose a Local Optimization (Local Opt.) method, which optimizes tokens within localized windows while leveraging contextual information to reduce error propagation. Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos. ReCo utilizes continuity loss to constrain representation changes, improving model robustness and reducing error accumulation. Extensive experiments on class- and text-to-video datasets demonstrate that our approach achieves superior performance to the baseline while halving the training cost without sacrificing quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。