将符号推理能力注入3D多模态大模型,实现可解释的复杂空间推理。
Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs
- 通过三阶段课程学习,将符号程序推理模式蒸馏到大模型中
- 在多个3D空间推理数据集上超越传统符号方法并匹配顶尖大模型表现
- 适合需要可解释性与开放词汇推理的研究者和应用开发者
当前3D空间推理方法面临根本权衡:神经符号3D(NS3D)概念学习器通过组合程序实现可解释推理,但受限于封闭词汇表和简单程序;端到端3D多模态大模型(3D MLLMs)可处理复杂自然语言和开放词汇概念,却缺乏显式空间验证的黑箱推理。我们提出APEIRIA,一种神经符号3D MLLM,通过将符号推理模式蒸馏到具有自然语言思维链的大模型中,融合两种范式。其三阶段课程逐步构建推理能力:a) 3D感知对齐将物体视觉-几何特征与大模型对齐;b) CoT-SFT从符号程序轨迹中学习查询分解与分步验证;c) CoT-RL将推理模式扩展至开放词汇概念和深层嵌套指令。通过迁移推理模式而非特定概念知识,APEIRIA保持了关键的NS3D优势:透明推理与规划/感知组件的模块化可替换性。在定位、问答和描述任务上的评估表明,APEIRIA优于先前的NS3D方法,并在3D空间推理数据集上达到最先进的3D MLLMs水平,统一了符号方法的系统性推理与大模型的灵活性。代码见https://github.com/oceanflowlab/APEIRIA。
原文摘要 · Abstract (English)
Current 3D spatial reasoning methods face a fundamental trade-off: neuro-symbolic 3D (NS3D) concept learners achieve interpretable reasoning through compositional programs but are constrained to closed-set concept vocabularies and simple programs; end-to-end 3D multi-modal LLMs (3D MLLMs) could handle complex natural language and open-vocabulary concepts but suffer from black-box reasoning without explicit spatial verification. We introduce APEIRIA, a neuro-symbolic 3D MLLM to bridge two paradigms by distilling symbolic reasoning patterns into MLLMs with natural language chain-of-thought. Our three-stage curriculum progressively builds reasoning capabilities: a) 3D perception alignment grounds object visual-geometric features to the LLM, b) CoT-SFT teaches query decomposition and stepwise verification from symbolic program traces, and c) CoT-RL extends reasoning patterns to open-set concepts and deeply nested instructions. By transferring reasoning patterns rather than concept-specific knowledge, APEIRIA preserves key NS3D virtues: transparent reasoning and modular interchangeability of planning and perception components. Evaluations on grounding, question answering, and captioning show that APEIRIA surpasses prior NS3D methods and matches state-of-the-art 3D MLLMs on 3D spatial reasoning datasets, unifying symbolic methods' systematic reasoning with MLLMs' flexibility. Code is available at https://github.com/oceanflowlab/APEIRIA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。