用文本指令驱动机器人分步塑形黏土，实现真实世界3D创作。
Planning and Reasoning with 3D Deformable Objects for Hierarchical Text-to-3D Robotic Shaping
- 先分块布局粗形，再逐步变形精修，分两阶段完成塑形。
- 仅凭文本提示即可生成多种简单3D形状，成功率高。
- 首个无需预设3D目标的端到端文本转3D塑形系统，适合机器人创意设计。
可变形物体操作是构建可部署于现实场景的自主机器人系统的关键挑战。本文以捏塑黏土为任务，提出首个从粗到精的自主塑形系统：首先将黏土块放置于工作区形成粗略轮廓，再通过一系列变形动作迭代优化形状。利用大语言模型生成子目标，训练基于点云区域的动作模型，从期望点云子目标预测机器人动作。本方法是首个无需提供显式3D目标或子目标的现实世界文本到3D塑形端到端流程。实验表明，系统仅通过文本提示即成功创建一组简单3D形状。此外，我们深入探讨了该任务的成功度量方式，对比了现有文本-图像与文本-点云相似性度量与人类评估结果。更多视频、人类评估细节及完整提示请见项目网站：https://sites.google.com/andrew.cmu.edu/hierarchicalsculpting
原文摘要 · Abstract (English)
Deformable object manipulation remains a key challenge in developing autonomous robotic systems that can be successfully deployed in real-world scenarios. In this work, we explore the challenges of deformable object manipulation through the task of sculpting clay into 3D shapes. We propose the first coarse-to-fine autonomous sculpting system in which the sculpting agent first selects how many and where to place discrete chunks of clay into the workspace to create a coarse shape, and then iteratively refines the shape with sequences of deformation actions. We leverage large language models for sub-goal generation, and train a point cloud region-based action model to predict robot actions from the desired point cloud sub-goals. Additionally, our method is the first autonomous sculpting system that is a real-world text-to-3D shaping pipeline without any explicit 3D goals or sub-goals provided to the system. We demonstrate our method is able to successfully create a set of simple shapes solely from text-based prompting. Furthermore, we explore rigorously how to best quantify success for the text-to-3D sculpting task, and compare existing text-image and text-point cloud similarity metrics to human evaluations for this task. For experimental videos, human evaluation details, and full prompts, please see our project website: https://sites.google.com/andrew.cmu.edu/hierarchicalsculpting
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。