融合图像与点云生成高精度3D汽车模型,提升自动驾驶仿真质量。
MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving

- 用多视角图像和激光点云联合引导生成,确保几何准确。
- 在Waymo数据集上生成车辆细节更丰富,纹理更真实。
- 适合自动驾驶仿真、训练数据生成等场景使用。
从自动驾驶场景中恢复逼真的3D车辆模型对于合成训练数据和构建仿真环境至关重要。然而,现有方法大多未能充分利用多模态传感器(如多视角图像和激光点云),且依赖神经渲染重建,导致网格质量较低。近期,原生3D生成模型取得显著进展,但通常不支持任意多视角输入,且对真实驾驶图像适应性差。本文提出MM-TRELLIS,一种面向真实场景的3D车辆生成方法,将自动驾驶数据集中的图像与点云融入原生3D生成模型。具体地,多视角图像作为条件输入,激光点云在测试时提供几何引导,确保结构准确性与跨视角一致性。去噪过程中,先将引导点云与模型先验对齐,再强制生成几何与点云一致。最后,引入基于3D高斯溅射透明度的体素过滤策略,消除噪声并生成干净网格。在Waymo数据集上的全面实验表明,该方法在高质量3D车辆生成方面优于现有方法。代码已开源:https://github.com/HongliXiao/MM-TRELLIS。
原文摘要 · Abstract (English)
Recovering realistic 3D vehicle models from autonomous driving scenes is crucial for synthesizing training data and building simulation environment. However, most existing vehicle generation methods fail to fully exploit multimodal sensors i.e. multi-view images and LiDAR point clouds) and rely on neural rendering based reconstruction, leading to low-quality mesh. Recently, native 3D generative models have made significant progress, yet they are not built for arbitrary multi-view inputs and often struggle with in-the-wild driving images. In this work, we present MM-TRELLIS, a multi-modal version of TRELLIS for in-the-wild 3D vehicle generation that integrates LiDAR and image sensors from autonomous driving datasets into native 3D generative models. Specifically, multi-view images are cycled as conditioning inputs, while LiDAR point clouds provide test-time guidance to ensure geometric accuracy and cross-view consistency. During denoising, we first align the guidance point cloud with the model priors, then enforce consistency between the generated geometry and the guidance point cloud. Finally, we introduce a voxel filtering strategy based on the opacity of 3D Gaussian Splatting to suppress floaters and produce clean meshes. Comprehensive experiments on Waymo dataset demonstrate our method outperforms existing methods in high-fidelity 3D vehicle generation. Code is available at https://github.com/HongliXiao/MM-TRELLIS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。