生成大规模动态城市4D场景,支持轨迹与指令驱动的智能渲染。
DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes
- 用六平面压缩+并行重建提升4D特征表示质量与效率。
- 在CarlaSC和Waymo上实现12.56% mIoU提升,训练速度翻倍。
- 支持多种条件生成,适合自动驾驶仿真与场景编辑研究。
城市场景生成近年来发展迅速,但现有方法多聚焦静态单帧场景,忽略了真实驾驶环境的动态特性。本文提出DynamicCity,一种新型4D占用生成框架,可生成大规模、高质量且带语义的动态4D场景。该框架包含两个核心模型:1)基于VAE的HexPlane压缩模型,采用新颖的投影模块将4D特征压缩为六个2D特征图,显著提升拟合质量(最高达12.56 mIoU提升);结合扩展与压缩策略并行重构3D特征体,相比逐点查询提升7.05 mIoU,训练速度提高2.06倍,内存减少70.84%;2)基于DiT的扩散模型,通过填充滚动操作将六平面重组为方形2D特征图,实现对HexPlane的高效生成。支持轨迹、命令、补全及布局条件等多种生成任务。在CarlaSC与Waymo数据集上的实验表明,DynamicCity在多项指标上显著优于现有最先进方法。代码与模型已开源,促进后续研究。
原文摘要 · Abstract (English)
Urban scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D occupancy generation framework capable of generating large-scale, high-quality dynamic 4D scenes with semantics. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D occupancy generation methods across multiple metrics. The code and models have been released to facilitate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。