用大模型生成可组装手工艺品，从照片中识别目标并匹配可用材料。
Prompt2Craft: Generating Functional Craft Assemblies with LLMs
- 基于输入照片提取可见部件，匹配模板网格并优化姿态。
- 将复杂形状简化为立方体、圆柱等基本几何体，提升可操作性。
- 通过比例搜索实现真实场景下的精准装配，适合机器人组装任务。
受传统手工技艺启发，我们正式提出「手工艺装配任务」：在未预先对应零件的情况下，利用现有物体精确复现目标物体。本文针对输入为野外拍摄的RGB图像的情况，首先使用掩码分割网络识别可见部件，再检索带标签的模板网格，并通过姿态优化确定最适配模板。随后将变形后的模板部件简化为立方体、圆柱等基本形状。最后设计搜索算法，依据局部与全局比例关系在场景中寻找对应关系。我们构建了遍历所有组合的基线方法，以前景图和掩码准确率等常见指标选出最优方案。实验表明，该方法在两个不同场景下表现接近基线，且实现了真实场景中的可行装配。
原文摘要 · Abstract (English)
Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labeled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。