让大模型直接理解3D模型的几何拓扑结构,实现文本与3D形状的精准对话。
BrepLLM: Enabling Large Language Models to Understand Boundary Representations
- 将B-rep模型转为融合几何拓扑信息的图结构,用层级编码器提取特征。
- 通过两阶段微调,使大模型在3D分类与描述任务上达到当前最优性能。
- 构建了26万对的Brep2Text数据集,适合3D生成、智能设计等场景研究者使用。
当前基于令牌序列的大语言模型难以直接处理包含复杂几何与拓扑信息的3D边界表示(B-rep)模型。为此,我们提出BrepLLM,首个使大模型直接解析与推理原始B-rep数据的多模态框架。BrepLLM采用两阶段训练流程:跨模态对齐预训练与双阶段微调。第一阶段设计自适应UV采样策略,将B-rep转化为融合几何与拓扑信息的图表示;随后构建分层BrepEncoder,从面、边等几何元素及拓扑中提取特征,生成全局标记与节点标记序列;通过对比学习,将全局标记与冻结的CLIP文本编码器(ViT-L/14)的文本嵌入进行初步对齐。第二阶段将预训练的BrepEncoder集成至LLM,采用渐进式双阶段策略对节点标记序列进行对齐:(1) 训练基于MLP的语义映射网络,利用2D视觉语言模型先验知识,将B-rep表示映射至2D视觉语义空间;(2) 使用LoRA实现对Q-Former与LLM主干网络的参数高效微调,最终达成3D-语言生成能力。此外,我们构建了包含269,444对B-rep与文本问答的Brep2Text数据集。实验表明,BrepLLM在3D物体分类与描述任务上达到当前最优表现。
原文摘要 · Abstract (English)
Current token-sequence-based Large Language Models (LLMs) struggle to directly process 3D Boundary Representation (B-rep) models that contain complex geometric and topological information. To this end, we propose BrepLLM, the first multimodal framework that enables LLMs to directly parse and reason over raw B-rep data. BrepLLM adopts a two-stage training pipeline: cross-modal alignment pre-training and two-stage LLM fine-tuning. In the first stage, we design an adaptive UV sampling strategy to convert B-reps into graph representations that integrate geometric and topological information. Subsequently, we construct a hierarchical BrepEncoder to extract features from geometric elements (faces and edges) and topology, generating a global token and a sequence of node tokens. Then, via contrastive learning, we conduct an initial alignment between this global token and the text embeddings of a frozen CLIP text encoder (ViT-L/14). In the second stage, we integrate the pre-trained BrepEncoder into the LLM and employ a two-stage progressive strategy to align the sequence of node tokens: (1) training an MLP-based semantic mapping network that utilizes the prior knowledge of a 2D-VLM to align the B-rep representation to the 2D visual semantic space; (2) utilizing LoRA for parameter-efficient fine-tuning of the Q-Former and the LLM backbone network to achieve the final 3D-language generation capability. Furthermore, we construct the Brep2Text dataset, which contains 269,444 B-rep and text question-answer pairs. Experiments demonstrate that BrepLLM achieves SOTA performance on 3D object classification and captioning tasks. The project page is available at https://user-deng.github.io/BrepLLM/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。