用帧重要性压缩上下文，让视频扩散模型能处理上千帧并防误差累积。
Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models
- 按帧重要性打包上下文，重要帧保留更长序列
- 支持千帧推理与大批次训练，减少生成偏差
- 适用于长视频生成，尤其适合需要稳定性的场景
我们提出一种神经网络结构 FramePack，用于训练下一帧（或下一帧段）预测的视频生成模型。FramePack 通过帧级重要性压缩输入帧上下文，使更多帧能在固定上下文长度内编码，重要帧享有更长上下文。帧重要性可基于时间接近度、特征相似度或混合度量。该压缩方法支持千帧级推理与较大批次训练。同时，我们提出漂移抑制方法以应对观测偏差（误差累积），包括早期建立终点、调整采样顺序和离散历史表示。消融实验验证了这些防漂移策略在单向视频流与双向视频生成中的有效性。最后，我们展示了现有视频扩散模型可通过 FramePack 进行微调，并分析了不同打包策略的差异。
原文摘要 · Abstract (English)
We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation. FramePack compresses input frame contexts with frame-wise importance so that more frames can be encoded within a fixed context length, with more important frames having longer contexts. The frame importance can be measured using time proximity, feature similarity, or hybrid metrics. The packing method allows for inference with thousands of frames and training with relatively large batch sizes. We also present drift prevention methods to address observation bias (error accumulation), including early-established endpoints, adjusted sampling orders, and discrete history representation. Ablation studies validate the effectiveness of the anti-drifting methods in both single-directional video streaming and bi-directional video generation. Finally, we show that existing video diffusion models can be finetuned with FramePack, and analyze the differences between different packing schedules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。