用空间感知掩码让大模型更好理解3D场景中的物体关系。
Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding
- 设计自适应掩码,根据物体空间分布而非顺序来决定注意力。
- 在多个3D场景语言任务中显著提升性能,最高提升12.7%。
- 无需修改模型结构,适合希望改进3D推理的开发者使用。
近期3D场景-语言理解研究利用大语言模型(LLM)将通用推理能力迁移至3D多模态场景。然而,现有方法普遍采用语言建模的标准解码器,依赖因果注意力掩码,导致两个根本性问题:对顺序无关的3D物体产生序列偏差,且限制物体与指令间的注意力交互,阻碍任务特定推理。为此,我们提出3D空间语言指令掩码(3D-SLIM),一种替代因果掩码的自适应注意力机制。3D-SLIM包含两个核心组件:基于几何密度的自适应掩码,按空间分布约束注意力而非词序;以及指令感知掩码,使物体标记可直接访问指令上下文。该设计使模型能依据空间关系处理物体,同时受用户任务引导。3D-SLIM实现简单,无需架构修改或新增参数,但在多个3D场景-语言任务上带来显著性能提升。跨多个基准和LLM基线的实验证明其有效性,凸显解码器设计在3D多模态推理中的关键作用。
原文摘要 · Abstract (English)
Recent advances in 3D scene-language understanding have leveraged Large Language Models (LLMs) for 3D reasoning by transferring their general reasoning ability to 3D multi-modal contexts. However, existing methods typically adopt standard decoders from language modeling, which rely on a causal attention mask. This design introduces two fundamental conflicts in 3D scene understanding: sequential bias among order-agnostic 3D objects and restricted object-instruction attention, hindering task-specific reasoning. To overcome these limitations, we propose 3D Spatial Language Instruction Mask (3D-SLIM), an effective masking strategy that replaces the causal mask with an adaptive attention mask tailored to the spatial structure of 3D scenes. Our 3D-SLIM introduces two key components: a Geometry-adaptive Mask that constrains attention based on spatial density rather than token order, and an Instruction-aware Mask that enables object tokens to directly access instruction context. This design allows the model to process objects based on their spatial relationships while being guided by the user's task. 3D-SLIM is simple, requires no architectural modifications, and adds no extra parameters, yet it yields substantial performance improvements across diverse 3D scene-language tasks. Extensive experiments across multiple benchmarks and LLM baselines validate its effectiveness and underscore the critical role of decoder design in 3D multi-modal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。