用视觉语言模型自动验证并修正3D建模代码，提升生成精度和成功率。
Generating CAD Code with Vision-Language Models for 3D Designs
- 通过视觉语言模型生成验证问题，迭代反馈修正3D建模代码
- 在GPT-4上使点云距离降低7.30%，程序成功率达5.0%提升
- 首个专用基准CADPrompt，支持自然语言到建模代码的评估
生成式AI已推动设计与制造领域实现高效自动化3D物体生成与修改。一种方法是利用大语言模型（LLM）生成计算机辅助设计（CAD）脚本代码，执行后渲染3D对象；但结果常不符合需求。由于3D对象（如形状、表面、尺寸）结构复杂，难以在代码中直接验证其正确性。本文提出CADCodeVerify，一种通过视觉语言模型（VLM）生成并回答一组验证问题，迭代反馈以验证生成对象并修正偏差的新方法。为评估该方法，我们构建了首个用于CAD代码生成的基准CADPrompt，包含200个自然语言提示及专家标注的3D对象脚本代码。实验表明，CADCodeVerify通过提供视觉反馈，提升了3D对象结构质量，并提高了程序编译成功率。应用于GPT-4时，其点云距离减少7.30%，程序成功率提升5.0%。
原文摘要 · Abstract (English)
Generative AI has transformed the fields of Design and Manufacturing by providing efficient and automated methods for generating and modifying 3D objects. One approach involves using Large Language Models (LLMs) to generate Computer- Aided Design (CAD) scripting code, which can then be executed to render a 3D object; however, the resulting 3D object may not meet the specified requirements. Testing the correctness of CAD generated code is challenging due to the complexity and structure of 3D objects (e.g., shapes, surfaces, and dimensions) that are not feasible in code. In this paper, we introduce CADCodeVerify, a novel approach to iteratively verify and improve 3D objects generated from CAD code. Our approach works by producing ameliorative feedback by prompting a Vision-Language Model (VLM) to generate and answer a set of validation questions to verify the generated object and prompt the VLM to correct deviations. To evaluate CADCodeVerify, we introduce, CADPrompt, the first benchmark for CAD code generation, consisting of 200 natural language prompts paired with expert-annotated scripting code for 3D objects to benchmark progress. Our findings show that CADCodeVerify improves VLM performance by providing visual feedback, enhancing the structure of the 3D objects, and increasing the success rate of the compiled program. When applied to GPT-4, CADCodeVerify achieved a 7.30% reduction in Point Cloud distance and a 5.0% improvement in success rate compared to prior work
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。