用单个增量令牌压缩视频,实现高效多样未来预测。
A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
- 将连续帧的特征差编码为单一增量令牌,大幅压缩视频表示
- 模型参数减少35倍以上,计算量降低2000倍,仍能生成多样未来
- 适合需要快速生成多路径预测的实时系统应用
预判多样的未来状态是视频世界建模的核心挑战。判别式模型生成确定性预测,隐含对可能未来的平均;现有生成式模型则计算成本高昂。近期研究表明,在视觉基础模型(VFM)的特征空间而非像素重建优化的潜空间中预测未来,可显著减少世界模型参数。然而多数方法仍为判别式。本文提出DeltaTok:将连续帧间VFM特征差编码为单一连续“增量”令牌;以及基于该令牌的生成式世界模型DeltaWorld,实现高效生成多样合理未来。增量令牌将三维时空视频转化为一维时间序列,例如在512x512帧下实现1024倍令牌压缩。该紧凑表示支持可行的多假设训练,多个未来并行生成,仅最佳结果受监督。推理时单次前向传播即可生成多样化预测。在密集预测任务上,DeltaWorld生成的未来更贴近真实结果,同时参数量超过35倍减少,浮点运算量减少2000倍。代码与权重:https://deltatok.github.io。
原文摘要 · Abstract (English)
Anticipating diverse future states is a central challenge in video world modeling. Discriminative world models produce a deterministic prediction that implicitly averages over possible futures, while existing generative world models remain computationally expensive. Recent work demonstrates that predicting the future in the feature space of a vision foundation model (VFM), rather than a latent space optimized for pixel reconstruction, requires significantly fewer world model parameters. However, most such approaches remain discriminative. In this work, we introduce DeltaTok, a tokenizer that encodes the VFM feature difference between consecutive frames into a single continuous "delta" token, and DeltaWorld, a generative world model operating on these tokens to efficiently generate diverse plausible futures. Delta tokens reduce video from a three-dimensional spatio-temporal representation to a one-dimensional temporal sequence, for example yielding a 1,024x token reduction with 512x512 frames. This compact representation enables tractable multi-hypothesis training, where many futures are generated in parallel and only the best is supervised. At inference, this leads to diverse predictions in a single forward pass. Experiments on dense forecasting tasks demonstrate that DeltaWorld forecasts futures that more closely align with real-world outcomes, while having over 35x fewer parameters and using 2,000x fewer FLOPs than existing generative world models. Code and weights: https://deltatok.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。