用RGB-D扫描生成可交互的逼真3D虚拟场景,支持物理渲染与动画。
LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans
- 基于结构化场景图解析场景,检索高质量艺术建模资产重建。
- 无需训练的物体检索在Scan2CAD上达到最优相似度,材料贴图抗遮挡光照差。
- 输出结果轻量可编辑,兼容主流图形管线,适合AR/VR、机器人等应用。
我们提出LiteReality,一个将室内环境的RGB-D扫描转换为紧凑、真实且可交互的3D虚拟复制品的新流程。LiteReality不仅重建视觉上接近现实的场景,还支持图形管线的关键特性——如物体独立性、可动性、高质量基于物理的渲染材质和物理交互。核心流程首先通过结构化场景图进行场景理解,解析出连贯的3D布局与物体;随后从精选资产库中检索最相似的3D艺术模型;接着通过材料绘制模块恢复空间变化的高质量材质;最后将重建场景集成至仿真引擎,赋予基础物理属性以实现交互行为。生成的场景紧凑、可编辑且完全兼容标准图形管线,适用于AR/VR、游戏、机器人及数字孪生。此外,LiteReality引入无训练物体检索模块,在Scan2CAD基准上达到顶尖相似度表现,并具备鲁棒的材料绘制能力,可将任意风格图像的外观转移至3D资产,即使存在严重错位、遮挡和弱光条件。我们在真实扫描与公开数据集上验证了该方法的有效性。
原文摘要 · Abstract (English)
We propose LiteReality, a novel pipeline that converts RGB-D scans of indoor environments into compact, realistic, and interactive 3D virtual replicas. LiteReality not only reconstructs scenes that visually resemble reality but also supports key features essential for graphics pipelines -- such as object individuality, articulation, high-quality physically based rendering materials, and physically based interaction. At its core, LiteReality first performs scene understanding and parses the results into a coherent 3D layout and objects with the help of a structured scene graph. It then reconstructs the scene by retrieving the most visually similar 3D artist-crafted models from a curated asset database. Next, the Material Painting module enhances realism by recovering high-quality, spatially varying materials. Finally, the reconstructed scene is integrated into a simulation engine with basic physical properties to enable interactive behavior. The resulting scenes are compact, editable, and fully compatible with standard graphics pipelines, making them suitable for applications in AR/VR, gaming, robotics, and digital twins. In addition, LiteReality introduces a training-free object retrieval module that achieves state-of-the-art similarity performance on the Scan2CAD benchmark, along with a robust material painting module capable of transferring appearances from images of any style to 3D assets -- even under severe misalignment, occlusion, and poor lighting. We demonstrate the effectiveness of LiteReality on both real-life scans and public datasets. Project page: https://litereality.github.io; Video: https://www.youtube.com/watch?v=ecK9m3LXg2c
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。