融合神经与物理渲染,实现自动驾驶仿真中动态物体自由放置与实时高精度生成。
Hybrid Rendering for Multimodal Autonomous Driving: Merging Neural and Physics-Based Simulation
- 用NeRF做教师模型,指导3DGS快速训练,兼顾泛化与实时性。
- 支持任意位置放置动态车辆,生成路面与车道线细节更真实,帧率可达实时。
- 适用于大规模场景仿真,适合自动驾驶研发与测试人员使用。
近年来,自动驾驶仿真中的神经重建模型取得了显著进展,动态模型日益普及。然而,这些模型通常仅限于处理遵循原始轨迹的域内物体。本文提出一种混合方法,结合神经重建与物理渲染的优势,实现虚拟放置传统网格动态代理至任意位置、调整环境条件及从新视角渲染。该方法显著提升新视角合成质量,尤其在道路表面和车道线表现上;通过创新的训练方法NeRF2GS,在保持交互帧率的同时完成大尺度(≥100,000平方米)重建。该方法利用噪声LiDAR点云提供的深度正则化训练定制化NeRF模型,并作为教师模型指导3DGS训练,确保深度、表面法向与相机外观建模准确。采用基于块的并行化训练策略,可输出分割掩码、表面法向图与深度图。仿真阶段支持基于光栅化的渲染后端(含深度合成与多相机模型)以实现实时相机模拟,以及基于光线追踪的后端用于精确LiDAR模拟。
原文摘要 · Abstract (English)
Neural reconstruction models for autonomous driving simulation have made significant strides in recent years, with dynamic models becoming increasingly prevalent. However, these models are typically limited to handling in-domain objects closely following their original trajectories. We introduce a hybrid approach that combines the strengths of neural reconstruction with physics-based rendering. This method enables the virtual placement of traditional mesh-based dynamic agents at arbitrary locations, adjustments to environmental conditions, and rendering from novel camera viewpoints. Our approach significantly enhances novel view synthesis quality -- especially for road surfaces and lane markings -- while maintaining interactive frame rates through our novel training method, NeRF2GS. This technique leverages the superior generalization capabilities of NeRF-based methods and the real-time rendering speed of 3D Gaussian Splatting (3DGS). We achieve this by training a customized NeRF model on the original images with depth regularization derived from a noisy LiDAR point cloud, then using it as a teacher model for 3DGS training. This process ensures accurate depth, surface normals, and camera appearance modeling as supervision. With our block-based training parallelization, the method can handle large-scale reconstructions (greater than or equal to 100,000 square meters) and predict segmentation masks, surface normals, and depth maps. During simulation, it supports a rasterization-based rendering backend with depth-based composition and multiple camera models for real-time camera simulation, as well as a ray-traced backend for precise LiDAR simulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。