纯视觉自动驾驶新框架，自监督生成3D占用图并高效建模场景。
RenderWorld: World Model with Self-Supervised 3D Label
- 用自监督高斯方法从图像生成3D占用标签，无需激光雷达。
- 基于高斯点云渲染，分割精度更高且显存消耗更低。
- 适合追求低成本高鲁棒性的自动驾驶系统研发者。
纯视觉端到端自动驾驶不仅比激光雷达与视觉融合更经济，而且比传统方法更可靠。为实现低成本、高鲁棒的纯视觉自动驾驶系统，我们提出RenderWorld：一个基于自监督高斯的图像到3D占用（Img2Occ）模块生成3D占用标签，通过AM-VAE分别编码空气与非空气区域，再利用世界模型进行预测与规划。该框架采用高斯点云（Gaussian Splatting）表示3D场景并渲染2D图像，相比基于NeRF的方法显著提升分割准确率并降低GPU内存占用。通过分离编码空气与非空气区域，实现更精细的场景表征，在4D占用预测和基于自回归世界模型的运动规划任务上达到当前最优性能。
原文摘要 · Abstract (English)
End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and uses world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, leading to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。