自动构建可交互的3D仿真环境,提升社交机器人训练真实性
D3D-GEN: Robot-Aware Domain-Grounded Interactive 3D World Generation for Social Robotics

- 用领域智能体+检索增强生成,自动收集并合成真实场景布局
- 无需固定模型库,支持医院/办公/住宅等多场景快速生成
- 适配Isaac Sim和Gazebo,适合做社交机器人仿真研究的团队
社交导航类具身智能的训练与验证高度依赖逼真的仿真环境,但现有方法难以兼顾真实感与可模拟性。本文提出D3D-GEN,一种结合领域智能体与检索增强生成(RAG)的新型世界生成系统,该系统基于特定领域构建持久化知识库。用户输入领域描述后,研究智能体自动采集公开的领域数据并构建数据库;随后通过RAG管道动态查询用户提供的语义数据库,生成合理的平面布局与物体摆放。整个过程无需依赖固定3D模型库,输出为可直接加载至Isaac Sim和Gazebo的完整交互式3D环境。我们已为室内住宅、医院、办公室等常见场景建立数据库,并为每类生成数十个不同且合理的仿真环境。系统配备本地Web前端,支持快速、交互式的世界生成。
原文摘要 · Abstract (English)
Training and validation of Embodied AI for social navigation critically depends on realistic simulation environments, yet many current approaches fail to find a balance between realism and simulability. We propose D3D-GEN, a novel world generation system that combines a domain agent with a retrieval-augmented generation (RAG) pipeline grounded in that domain. Our system enables users to rapidly generate domain-grounded, fully interactive 3D worlds by automating both the collection of domain knowledge and the synthesis of realistic floorplans and object placements, without dependence on any fixed 3D model database. Given a domain description prompt, the research agent collects publicly accessible domain-specific data and constructs a persistent domain database. Using this database, our RAG pipeline generates plausible floorplans and object placements by dynamically querying a user-provided semantic database, which can be easily extended or modified. The output is a fully interactive 3D world loadable by the popular simulators Isaac Sim and Gazebo. With our approach, we have built databases for several common domains (indoor residential, hospital, office) and generated dozens of distinct, plausible simulation environments for each domain. We present D3D-GEN with a local web frontend that facilitates rapid, interactive world generation for robot simulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。