让3D大模型能直接命名和推理物体部件，仅用不到100万参数。
3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
- 用可学习标记和预留词表将点云分块，使部件可被语言模型直接引用。
- 在PartVerse-QA上达到13.78%精确匹配率，优于现有模型3.03分（GPT-4o评分）。
- 仅新增不足100万参数，无需分割头或框式结构，适合资源受限场景。
3D多模态大语言模型通常将3D物体整体描述，无法命名或推理其部件。以往方法通过增加分割解码器、更重的3D编码器或边界框语法，带来显著参数开销。本文提出全新路径：重构输入令牌流，使部件可通过语言模型自身词汇直接访问。3D-PLOT-LLM将冻结点编码器的补丁划分为K个局部连贯区域，在每个区域前插入可学习的区域标记与预留词 <part_k>；标记空间优化模块（MSR）基于区域空间统计与邻接关系对每个标记进行条件化。模型因此可在输出中引用部件，并响应以部件为指代的提示，这是此前对象级3D MLLMs不具备的能力。为验证该接口，我们构建了基于PartVerse网格标注的词汇级部件问答基准PartVerse-QA（77K训练对，588个独立对象测试查询），3D-PLOT-LLM在该任务上实现0.459的标题到槽位Jaccard值与13.78%的精确匹配率，槽位到标题的GPT-4o评分达44.68。在3DCoMPaT-GrIn部件感知定位描述任务中，3D-PLOT-LLM在所有文本指标上超越PointLLM、Kestrel、PARIS3D、SegPoint，且在3/4项上优于ShapeLLM，最高领先3.03分（GPT-4o）。在Objaverse全对象描述任务中，第二阶段引入PartVerse-QA后，相比PointLLM提升0.65 SBERT与1.85 GPT-4o，且在5项传统指标中的4项超越PointLLM-PiSA，尽管目标为部件定位而非整体描述。全部过程仅新增不足100万可训练参数，远低于先前部件感知模型，且无需分割解码器或边界框头。
原文摘要 · Abstract (English)
3D multimodal large language models (3D MLLMs) describe a 3D object as a whole but cannot address, name, or reason about its parts. Prior part-aware attempts add segmentation decoders, heavier 3D encoders, or bounding-box grammars at substantial parameter cost. We take a fundamentally different path: we reorganize the input token stream so that parts become directly addressable through the LLM's own vocabulary. Our model, 3D-PLOT-LLM, partitions the frozen point encoder's patches into K locally coherent regions and inserts, before each region's patch tokens, a learnable per-region marker and a reserved vocabulary token <part_k>; a Marker-Space Refinement (MSR) module then conditions each marker on its region's spatial statistics and adjacency neighbors. The model thus cites parts in its output and follows prompts that refer to parts by token, a capability absent from prior object-level 3D MLLMs. To probe this interface, we construct PartVerse-QA, a vocabulary-level part-QA benchmark adapted from PartVerse mesh annotations (77K training pairs and 588 held-out queries on disjoint object splits), on which 3D-PLOT-LLM reaches caption-to-slots Jaccard 0.459 and Exact-match 13.78%, with a slot-to-caption GPT-4o judge of 44.68. On the 3DCoMPaT-GrIn part-aware grounded description benchmark, 3D-PLOT-LLM outperforms PointLLM, Kestrel, PARIS3D, and SegPoint on every text-output metric, and ShapeLLM on 3 of 4, with up to +3.03 GPT-4o judge over PointLLM. On Objaverse whole-object captioning, adding PartVerse-QA at Stage 2 yields +0.65 SBERT and +1.85 GPT-4o over PointLLM, and tops PointLLM-PiSA on 4 of 5 traditional metrics (SBERT, SimCSE, BLEU-1, METEOR) despite targeting a different (part-grounded) objective. All with under 1M new trainable parameters on a frozen point encoder, an order of magnitude below prior part-aware 3D MLLMs, and no segmentation decoder or bounding-box head.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。