arXiv:2606.05445cs.AI2026-06

用多模态大模型组装积木,提升视觉与空间推理能力。

Brick-Composer: Using MLLMs for Assembly with Diverse Bricks

论文配图:Brick-Composer: Using MLLMs for Assembly with Diverse Bricks
图 1 · 摘自论文原文
  • 将积木组装建模为分步决策任务,包含选块与姿态预测。
  • 新框架使选块准确率提升3倍,步骤级成功率达15%。
  • 适合研究具身智能、物理推理与多模态模型应用的学者。

我们设想能够理解任意设计并用可复用积木构建真实物体的AI代理。作为迈向这一愿景的第一步,我们探究多模态大语言模型(MLLMs)是否具备积木组装所需的视觉定位与空间推理能力。将积木组装视为序列决策问题,每一步包含两个子任务:积木选择(从候选组件中识别目标积木)和积木姿态估计(预测所选积木的位置与朝向)。为此,我们提出BC-Bench(积木构造基准),首个用于评估MLLMs在多样化积木组装上表现的基准。实验表明,当前最先进MLLMs仍远未达到可靠建造者水平,难以完成细粒度积木选择,且姿态估计精度不足。为弥补此差距,我们提出Brick-Composer学习框架,通过三种互补信号增强MLLMs的装配能力:人类设计启发(提供富含功能提示的构造示范)、世界反馈(将预测动作与视觉及物理后果对齐)、合成经验(扩展学习至现有设计之外)。Brick-Composer使积木选择准确率提升超三倍,显著降低姿态估计误差,并将严格步骤级组装成功率从不足1%提升至约15%。训练后,Qwen-3-8B可正确完成高达42%的完整对象组装步骤,表明通过针对性、物理对齐的学习,MLLMs可获得装配能力。

原文摘要 · Abstract (English)

We dream of AI agents that can read arbitrary designs and construct real-world objects from reusable building blocks. As a first step toward this vision, we study whether multimodal large language models (MLLMs) possess the visual grounding and spatial reasoning capabilities required for brick assembly. We formulate brick assembly as a sequential decision-making problem, where each step involves two subtasks: brick selection, identifying the target brick from candidate components, and brick pose estimation, predicting where and how the selected brick should be placed. To support this study, we introduce BC-Bench (Brick Construction Benchmark), the first benchmark for evaluating MLLMs on assembly with diverse bricks. Experiments show that current state-of-the-art MLLMs remain far from reliable builders, struggling with fine-grained brick selection and failing at precise pose estimation. To bridge this gap, we propose Brick-Composer, a learning framework that equips MLLMs with assembly skills through three complementary signals: Human Design Sparks, which provide affordance-rich construction demonstrations; World Feedback, which grounds predicted actions in visual and physical consequences; and Synthetic Experience, which scales learning beyond existing object designs. Brick-Composer improves brick selection accuracy by over three times, substantially reduces pose estimation errors, and raises strict step-level assembly success from less than 1% to around 15%. After training, a Qwen-3-8B can correctly compose up to 42% of the steps for a complete object, suggesting that MLLMs can acquire assembly capabilities through targeted, physically grounded learning.

多模态模型积木组装具身智能空间推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。