用多视角选图+语言提示,精准定位3D模型的边、面等几何实体。
MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes

- 根据语言查询动态选最易识别的视角,提升感知效果。
- 面级交并比提升1.7倍,边级精度提升超4.5倍。
- 适合需要精细3D几何理解的工业设计与机器人场景。
精确识别和定位3D物体中的边、平面区域及曲面等几何实体,是计算机辅助设计(CAD)、机器人操作和科学模拟的基础。尽管现代视觉语言模型(VLMs)在图像领域实现了先进的指代分割(RIS),但将此类语言驱动定位扩展到结构化3D几何仍面临挑战:3D对象外观对视角高度敏感,单一视角可能使目标实体清晰可见,而另一视角则因严重遮挡或透视变形导致难辨。本文提出MV-GEL(多视角几何实体定位)框架,基于自然语言查询在多边形网格上定位细粒度几何实体。核心思想是:可靠的CAD实体(面、边或体)定位依赖于选择能最大化目标实体可读性的视角。我们引入GELviews——一个基于语言提示的视角排序模块,按语言引导的可观测性优先筛选视角。选定视角经由基于VLM的推理分割主干处理,预测掩码通过几何感知射线投射映射至对应网格。该框架完全不依赖特定CAD格式,仅需3D网格数据。实验表明,相比基线模型,面级交并比提升1.7倍,边级F1值提升超过4.5倍,显著优于基于CLIP的方法和随机视角采样,尤其在薄型和视角敏感结构上表现突出。代码、数据集及训练权重已公开于https://github.com/kbali1297/MV-GEL。
原文摘要 · Abstract (English)
Identifying and grounding precise geometric entities, such as edges, planar regions, and curved surfaces within 3D objects, is foundational to computer-aided design (CAD), robotic manipulation, and scientific simulation. Although modern Vision Language Models (VLMs) have advanced referring segmentation (RIS) in the image domain, extending such language-driven localization to structured 3D geometry is substantially harder. The 3D object appearance is highly sensitive to viewpoints; a single perspective may render a target entity clearly observable, while another may suffer from severe occlusion or foreshortening. In this work, we attempt to solve these challenges with MV-GEL (Multi-View Geometric Entity Localization), a framework for localizing fine-grained geometric entities on polygon meshes from natural language queries. Our key insight is that reliable CAD entity (i.e., faces, edges or solids) localization depends on selecting views that make the queried entity maximally interpretable. We introduce GELviews, a prompt-conditioned ranking module that prioritizes viewpoints based on language prompted observability of geometric CAD entities. Selected views are processed by a VLM-based reasoning segmentation backbone, and predicted masks are lifted to the corresponding meshes via geometry-aware ray casting. Our framework is completely CAD agnostic and relies only on 3D meshes. Experiments show up to a 1.7X improvement in face-level IoU and over 4.5X gains in edge-level F1 compared to vanilla baselines, substantially outperforming CLIP-based and random view sampling, particularly for thin and view-sensitive structures. The dataset, code and trained checkpoints are available at https://github.com/kbali1297/MV-GEL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。