仅用摄像头实现高保真3D数字孪生重建，无需复杂标定
Material-informed Gaussian Splatting for 3D World Reconstruction in a Digital Twin
- 基于多视角图像的3D高斯点云重建，融合视觉模型提取材质掩码
- 将材质标签投影到网格表面，赋予物理属性实现精准传感器模拟
- 在真实车载数据上验证，效果媲美激光雷达-相机融合方案
数字孪生中的3D重建通常依赖激光雷达，虽几何精度高但缺乏语义与纹理。传统激光雷达-相机融合需复杂标定，且对玻璃等材料表现不佳。本文提出仅用摄像头的重建流程：从多视角图像生成3D高斯点云，通过视觉模型提取语义材质掩码，将高斯表示转换为带材质标签的网格表面，并赋予基于物理的材质属性，以支持现代图形引擎和仿真器中的高保真传感器模拟。该方法结合了照片级重建与物理材质赋值，在内部测试车辆数据集上验证，以激光雷达为真实反射率参考，同时使用图像相似性指标评估，结果表明其传感器模拟精度可媲美激光雷达-相机融合方案，且无需硬件标定。
原文摘要 · Abstract (English)
3D reconstruction for Digital Twins often relies on LiDAR-based methods, which provide accurate geometry but lack the semantics and textures naturally captured by cameras. Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass, which are visible in images but poorly represented in point clouds. We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images, extracts semantic material masks via vision models, converts Gaussian representations to mesh surfaces with projected material labels, and assigns physics-based material properties for accurate sensor simulation in modern graphics engines and simulators. This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion while eliminating hardware complexity and calibration requirements. We validate our camera-only method using an internal dataset from an instrumented test vehicle, leveraging LiDAR as ground truth for reflectivity validation alongside image similarity metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。