用自然语言生成多部件机器人装配模型，提升设计自动化与人机协作效率。
Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models
- 结合3D生成AI与视觉语言模型，实现从文本到多组件结构的零样本分解。
- 用户偏好测试中VLM方案获90.6%认可，远超规则与随机分配。
- 支持对话式反馈修正组件分配，增强人类对生成物体的控制力。
3D生成AI的进步使仅凭文本提示即可创建实物，但在涉及多种组件类型时仍面临挑战。本文提出一种集成3D生成AI与视觉语言模型（VLMs）的流水线，实现从自然语言指令驱动多组件物体的机器人装配。该方法利用VLM进行零样本、多模态的几何与功能推理，将AI生成的网格分解为包含预定义结构与面板组件的多组件3D模型。实验表明，VLM能根据物体几何与功能判断哪些区域需添加面板组件。在测试对象评估中，用户对VLM生成的组件分配选择率达90.6%，显著优于基于规则的59.4%和随机分配的2.5%。系统还支持通过对话反馈动态优化组件配置，提升用户在生成式AI与机器人协同造物过程中的控制权与参与感。
原文摘要 · Abstract (English)
Advances in 3D generative AI have enabled the creation of physical objects from text prompts, but challenges remain in creating objects involving multiple component types. We present a pipeline that integrates 3D generative AI with vision-language models (VLMs) to enable the robotic assembly of multi-component objects from natural language. Our method leverages VLMs for zero-shot, multi-modal reasoning about geometry and functionality to decompose AI-generated meshes into multi-component 3D models using predefined structural and panel components. We demonstrate that a VLM is capable of determining which mesh regions need panel components in addition to structural components, based on the object's geometry and functionality. Evaluation across test objects shows that users preferred the VLM-generated assignments 90.6% of the time, compared to 59.4% for rule-based and 2.5% for random assignment. Lastly, the system allows users to refine component assignments through conversational feedback, enabling greater human control and agency in making physical objects with generative AI and robotics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。