让大模型像解题一样逐步推理3D点云，避免乱猜。
PointCoT: A Multi-modal Benchmark for Explicit 3D Geometric Reasoning
- 引入显式思维链，先分析几何结构再回答
- 在8.6万条标注数据上训练，准确率领先
- 适合需要精确3D理解的自动驾驶等场景
虽然多模态大模型在2D场景中表现优异，但将其感知智能扩展到3D点云理解仍面临挑战。现有方法主要聚焦于将3D特征与预训练模型对齐，却通常将几何推理视为隐式映射过程，跳过中间逻辑步骤，导致几何幻觉。模型常自信生成看似合理但缺乏精确结构支撑的答案。为弥补这一差距，我们提出PointCoT，一种支持多模态大模型进行显式思维链（CoT）推理的新型框架。该框架倡导“看、想、答”范式，通过监督模型生成基于几何事实的推理过程后再预测答案。为此，我们构建了包含约86,000条指令微调样本的Point-Reason-Instruct大规模基准，具备分层思维链标注。借助双流多模态架构，方法融合语义外观与几何真实。大量实验表明，PointCoT在复杂推理任务上达到当前最优性能。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) demonstrate proficiency in 2D scenes, extending their perceptual intelligence to 3D point cloud understanding remains a significant challenge. Current approaches focus primarily on aligning 3D features with pre-trained models. However, they typically treat geometric reasoning as an implicit mapping process. These methods bypass intermediate logical steps and consequently suffer from geometric hallucinations. They confidently generate plausible responses that fail to ground in precise structural details. To bridge this gap, we present PointCoT, a novel framework that empowers MLLMs with explicit Chain-of-Thought (CoT) reasoning for 3D data. We advocate for a \textit{Look, Think, then Answer} paradigm. In this approach, the model is supervised to generate geometry-grounded rationales before predicting final answers. To facilitate this, we construct Point-Reason-Instruct, a large-scale benchmark comprising $\sim$86k instruction-tuning samples with hierarchical CoT annotations. By leveraging a dual-stream multi-modal architecture, our method synergizes semantic appearance with geometric truth. Extensive experiments demonstrate that PointCoT achieves state-of-the-art performance on complex reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。