让AI按步骤生成复杂物体,看得见每一步的组装逻辑。
Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thought
- 用视觉思维链逐步生成物体,边画边规划,不依赖外部工具。
- 在26000条组装轨迹上训练,组件数量识别准确率达88.4%。
- 适合需要精确结构控制的工业设计、三维建模场景。
文本到图像生成模型虽具高视觉保真度,但在组合结构约束下仍显脆弱,尤其体现在生成计数、属性绑定和部件级关系方面。为此,我们提出视觉思维链框架 Shape-of-Thought (SoT),实现无需推理时外部引擎支持的2D渲染域中渐进式形状组装。SoT 训练统一的多模态自回归模型,交替生成文本计划与中间渲染状态,使模型在无显式几何表示的情况下掌握组装逻辑。与纯文本思维链不同,每一步决策均基于可视状态,可追踪计数、连接、拓扑及部件添加错误。为支持该范式,我们构建了基于零件级CAD层级的大型数据集 SoT-26K 及评估结构完整性和轨迹忠实性的基准 T2S-CompBench。在 SoT-26K 上微调后,组件计数准确率达88.4%,结构拓扑达84.8%,较直接生成分别提升24.2和19.3个百分点。SoT 建立了一个透明的渲染域结构感知生成测试平台。代码已开源。
原文摘要 · Abstract (English)
Multimodal models for text-to-image generation have achieved strong visual fidelity, yet they remain brittle under compositional structural constraints, notably generative numeracy, attribute binding, and part-level relations. To address these challenges, we propose Shape-of-Thought (SoT), a visual CoT framework for process-supervised progressive shape assembly in the rendered 2D domain, without external engines at inference time. SoT trains a unified multimodal autoregressive model to generate interleaved textual plans and rendered intermediate states, helping the model capture shape-assembly logic without producing explicit geometric representations. Unlike text-only CoT, each decision is grounded in a rendered state, making counts, attachments, topology, and intermediate part-addition errors inspectable across the trajectory. To support this paradigm, we introduce SoT-26K, a large-scale dataset of grounded assembly traces derived from part-based CAD hierarchies, and T2S-CompBench, a benchmark for evaluating structural integrity and trace faithfulness. Fine-tuning on SoT-26K achieves 88.4% on component numeracy and 84.8% on structural topology, outperforming direct generation by +24.2 points on component numeracy and +19.3 points on structural topology. SoT establishes a transparent testbed for rendered-domain structure-aware generation. The code is available at https://github.com/yuhuo03/Shape-of-Thought.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。