让大模型零训练生成3D草图,靠对比反馈自我提升
3DrawAgent: Teaching LLM to Draw in 3D with Early Contrastive Experience
- 用语言驱动逐段画3D贝塞尔曲线,结合感知与语义评估优化
- 在10个测试场景中生成复杂连贯3D草图,对新形状有良好泛化能力
- 无需参数更新,通过对比经验自进化,适合创意设计与交互应用
在三维空间中绘图有助于对形状、结构和空间关系进行表达性推理,但通过自然语言生成3D草图仍是一大挑战。本文提出3DrawAgent,一种无需训练、基于语言的3D草图生成框架,利用大语言模型(LLM)在几何反馈下逐段绘制3D贝塞尔曲线。不同于以往2D草图代理,该方法引入相对经验优化策略,适配最新提出的组奖励策略优化(GRPO)范式。不依赖显式真实标签,而是构建生成草图之间的成对比较,每对包含基于CLIP感知奖励和LLM细粒度定性评估的相对优劣结果。这些经验用于迭代优化3D绘图先验知识,实现模型空间认知的黑箱强化。该设计使模型可在不更新参数的情况下自我改进空间理解与绘图质量。实验表明,3DrawAgent可从多样化文本提示生成复杂连贯的3D贝塞尔草图,展现出涌现的几何推理能力,并对新形状具有良好泛化性能,为无训练3D草图智能开辟新范式。
原文摘要 · Abstract (English)
Sketching in 3D space enables expressive reasoning about shape, structure, and spatial relationships, yet generating 3D sketches through natural language remains a major challenge. In this work, we introduce 3DrawAgent, a training-free, language-driven framework for 3D sketch generation that leverages large language models (LLMs) to sequentially draw 3D Bezier curves under geometric feedback. Unlike prior 2D sketch agents, our method introduces a relative experience optimization strategy that adapts the recently proposed Group Reward Policy Optimization (GRPO) paradigm. Instead of relying on explicit ground-truth supervision, we construct pairwise comparisons among generated sketches, with each pair consisting of a relatively better and a worse result based on CLIP-based perceptual rewards and LLM-based fine-grained qualitative assessment. These experiences are then used to iteratively refine the prior knowledge of 3D drawing, enabling black-box reinforcement of the model's 3D awareness. This design allows our model to self-improve its spatial understanding and drawing quality without parameter updates. Experiments show that 3DrawAgent can generate complex and coherent 3D Bezier sketches from diverse textual prompts, exhibit emergent geometric reasoning, and generalize to novel shapes, establishing a new paradigm for advancing the field of training-free 3D sketch intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。