用少量参数实现高效3D CAD模型生成，视觉文本信息融合更轻量。
GuideCAD: A Lightweight Multimodal Framework for 3D CAD Model Generation via Prefix Embedding
- 通过前缀嵌入将图像特征映射到语言模型，融合视觉与文本信息
- 仅用四分之一参数量，训练效率提升一倍，生成质量相当
- 适合需要快速部署、资源受限的3D建模应用
现有多模态3D CAD生成方法计算开销大，需高效训练方案。为此，我们提出GuideCAD，利用语义丰富的视觉-文本表示，仅用少量可训练参数生成3D CAD模型。具体而言，GuideCAD通过映射网络将图像嵌入转换为前缀嵌入，使预训练大语言模型能整合视觉与文本信息。基于此，一个Transformer解码器使用视觉-文本嵌入预测构建序列，生成3D CAD模型。为评估，我们构建新数据集GuideCAD，包含文本-图像对，每对包含描述3D CAD构建序列的文本提示及其对应的3D CAD图像。实验表明，GuideCAD在生成质量相当的情况下，参数量约为微调方法的四分之一，训练效率提高一倍。代码与数据集已开源。
原文摘要 · Abstract (English)
Multi-modal approaches used for 3D CAD generation require substantial computational resources, necessitating efficient training. To address this, we propose GuideCAD, which leverages semantically rich visual-textual representations having only a small number of trainable parameters to generate 3D CAD models. Specifically, GuideCAD uses a mapping network that converts image embeddings into prefix embeddings, enabling a pretrained large language model to integrate visual and textual information. As a result, a transformer-based decoder predicts the construction sequence using the visual-textual embeddings in order to generate the 3D CAD model. For experimental evaluation, we construct a new dataset, referred to as GuideCAD, which consists of text-image pairs. Each pair includes a text prompt that represents a 3D CAD construction sequence and its corresponding 3D CAD image. Our experimental results show that GuideCAD generates comparably high-quality 3D CAD models while using approximately four times fewer parameters and achieving twice the training efficiency compared to fine-tuning approaches. We have released the source code and dataset for our method at: https://github.com/mskimS2/GuideCAD
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。