提出新型4D场景预测框架,高效压缩3D场景并保持动态细节。
$I^{2}$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting
- 分内外两阶段令牌化:内场景多尺度残差量化,外场景残差时序聚合
- 4D占用率预测mIoU提升25.1%,推理速度达37.0 FPS,仅需2.9GB显存
- 适合自动驾驶场景生成与实时动态建模,尤其关注高效率与细节保留
基于占据的场景模型在应对自动驾驶中的边缘案例方面具有巨大潜力。尽管令牌化已革新图像与视频生成,但复杂3D场景的高效令牌化仍是3D世界模型的关键挑战。为此,我们提出$I^{2}$-World,一种高效的4D占据预测框架。该方法将场景令牌化解耦为场景内与场景间两个令牌器:场景内令牌器采用多尺度残差量化策略,分层压缩3D场景并保留空间细节;场景间令牌器则残差聚合时间维度上的依赖关系。这一双设计在保持3D令牌器紧凑性的同时,维持了4D令牌器的动态表达能力。不同于解码器单向的GPT式自回归模型,$I^{2}$-World采用编码器-解码器架构:编码器从当前场景聚合空间上下文,并预测变换矩阵以实现对场景生成的高层控制;解码器在该矩阵与历史令牌条件下生成,确保生成过程的时间一致性。实验表明,$I^{2}$-World在4D占据预测上达到顶尖性能,相比现有方法提升25.1%的mIoU与36.9%的IoU,同时具备卓越计算效率:训练仅需2.9 GB显存,推理可达37.0 FPS。代码已开源于https://github.com/lzzzzzm/II-World。
原文摘要 · Abstract (English)
Forecasting the evolution of 3D scenes and generating unseen scenarios via occupancy-based world models offers substantial potential for addressing corner cases in autonomous driving systems. While tokenization has revolutionized image and video generation, efficiently tokenizing complex 3D scenes remains a critical challenge for 3D world models. To address this, we propose $I^{2}$-World, an efficient framework for 4D occupancy forecasting. Our method decouples scene tokenization into intra-scene and inter-scene tokenizers. The intra-scene tokenizer employs a multi-scale residual quantization strategy to hierarchically compress 3D scenes while preserving spatial details. The inter-scene tokenizer residually aggregates temporal dependencies across timesteps. This dual design preserves the compactness of 3D tokenizers while retaining the dynamic expressiveness of 4D tokenizers. Unlike decoder-only GPT-style autoregressive models, $I^{2}$-World adopts an encoder-decoder architecture. The encoder aggregates spatial context from the current scene and predicts a transformation matrix to enable high-level control over scene generation. The decoder, conditioned on this matrix and historical tokens, ensures temporal consistency during generation. Experiments demonstrate that $I^{2}$-World achieves state-of-the-art performance, outperforming existing methods by 25.1\% in mIoU and 36.9\% in IoU for 4D occupancy forecasting while exhibiting exceptional computational efficiency: it requires merely 2.9 GB of training memory and achieves real-time inference at 37.0 FPS. Our code is available on https://github.com/lzzzzzm/II-World.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。