用语义对齐机制让大模型精准分割3D物体,提升准确率。
Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model
- 设计对象中心表征,让语言模型理解3D物体语义与关系。
- 在ScanNetv2上提升7.3点mIoU,Multi3DRefer上提升6.0点。
- 适合做3D语义理解、物体分割与多模态交互的研究者。
基于大语言模型(LLM)的3D物体分割因具备广泛语义、任务灵活与强泛化能力成为主流方法。然而,该范式受制于表征错位:LLM处理高层语义令牌,而3D点云仅表达密集几何结构。此前方法中,错位限制了输入与输出阶段。输入时,密集点块需大量预对齐,削弱物体级语义并混淆相似干扰项;输出时,预测仅依赖密集特征,缺乏显式几何线索,导致细粒度精度损失。为此,我们提出点语言模型(PLM),一种无需大规模3D-文本或3D-图像预对齐的通用框架。引入面向对象的判别性表征(OcDR),通过硬负样本感知训练目标学习捕捉目标语义与场景关系的对象中心令牌,缓解LLM令牌与3D点之间的错位,增强对干扰项的鲁棒性,并促进LLM内的语义级推理。为实现精准分割,提出几何重激活解码器(GRD),通过结合携带LLM推断几何的OcDR令牌与对应密集特征进行掩码预测,全程保留完整密集特征。大量实验表明,PLM在ScanNetv2上实现+7.3 mIoU,在Multi3DRefer上实现+6.0 mIoU,跨7个基准、4类任务均取得一致提升,证明全面对象中心推理对稳健3D理解的有效性。
原文摘要 · Abstract (English)
3D object segmentation with Large Language Models (LLMs) has become a prevailing paradigm due to its broad semantics, task flexibility, and strong generalization. However, this paradigm is hindered by representation misalignment: LLMs process high-level semantic tokens, whereas 3D point clouds convey only dense geometric structures. In prior methods, misalignment limits both input and output. At the input stage, dense point patches require heavy pre-alignment, weakening object-level semantics and confusing similar distractors. At the output stage, predictions depend only on dense features without explicit geometric cues, leading to a loss of fine-grained accuracy. To address these limitations, we present the Point Linguist Model (PLM), a general framework that bridges the representation gap between LLMs and dense 3D point clouds without requiring large-scale pre-alignment between 3D-text or 3D-images. Specifically, we introduce Object-centric Discriminative Representation (OcDR), which learns object-centric tokens that capture target semantics and scene relations under a hard negative-aware training objective. This mitigates the misalignment between LLM tokens and 3D points, enhances resilience to distractors, and facilitates semantic-level reasoning within LLMs. For accurate segmentation, we introduce the Geometric Reactivation Decoder (GRD), which predicts masks by combining OcDR tokens carrying LLM-inferred geometry with corresponding dense features, preserving comprehensive dense features throughout the pipeline. Extensive experiments show that PLM achieves significant improvements of +7.3 mIoU on ScanNetv2 and +6.0 mIoU on Multi3DRefer for 3D referring segmentation, with consistent gains across 7 benchmarks spanning 4 different tasks, demonstrating the effectiveness of comprehensive object-centric reasoning for robust 3D understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。