无需训练,用文字或图片自动生成可交互的桌面操作场景
TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation
- 通过生成式实例提取获取独立3D物体模型
- 采用可微旋转优化与俯视空间对齐,实现无碰撞布局
- 生成物理真实的仿真环境,支持零样本真实机械臂迁移
仿真为大规模机器人操作数据采集提供了低成本、可扩展的路径。然而,现有3D场景生成方法难以直接用于操作数据合成,因其生成场景常缺乏实例级可交互性与物理合理性。针对桌面操作任务,我们提出TabletopGen,一种无需训练、全自动的桌面场景生成与交互仿真引擎。从文本或单张图像出发,首先通过生成式实例提取获得独立3D物体模型;其次,引入新型姿态与尺度对齐方法,利用可微旋转优化器和俯视空间对齐机制恢复无碰撞场景布局;最后,在物理模拟器中组装生成场景并添加碰撞几何,得到稳定可交互的环境,用于合成多模态操作数据。大量实验与用户研究证明,TabletopGen在视觉保真度、布局准确性和物理合理性方面达到当前最优水平。此外,通过零样本真实-仿真-真实策略迁移验证了所收集轨迹在真实机械臂上的可执行性,表明TabletopGen可作为可靠的机器人操作数据合成引擎。
原文摘要 · Abstract (English)
Simulation provides a low-cost, scalable pathway to large-scale robotic manipulation data collection. However, existing 3D scene generation methods can rarely be applied directly to manipulation data synthesis, as their generated scenes often lack instance-level interactivity and physical plausibility. Focusing on tabletop manipulation, we propose TabletopGen, a training-free and automated tabletop scene generation and interactive simulation engine. Starting from text or a single image, we first obtain independent 3D object models via generative instance extraction. Second, we introduce a novel pose and scale alignment approach that recovers a collision-free scene layout using a Differentiable Rotation Optimizer and a Top-View Spatial Alignment mechanism. Finally, we assemble the generated scene in a physics simulator with collision geometry, yielding a stable, interactable environment for synthesizing multimodal manipulation data. Extensive experiments and user studies demonstrate that TabletopGen achieves state-of-the-art performance in visual fidelity, layout accuracy, and physical plausibility. Furthermore, we validate the executability of the collected trajectories on a real robotic arm via zero-shot real-to-sim-to-real policy transfer, indicating that TabletopGen can serve as a reliable data engine for robotic manipulation data synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。