PerLA让大模型更懂3D世界，兼顾局部细节与全局上下文。
PerLA: Perceptive 3D Language Assistant
- 用希尔伯特曲线保局部结构，通过跨注意力融合细节与全局信息
- 在ScanQA上问答性能提升1.34点，密集描述任务最高增4.22点
- 适合需要精准3D理解的视觉语言模型研究者
让大型语言模型理解三维物理世界是新兴且具有挑战性的研究方向。当前处理点云的方法通常对场景进行下采样或分块分析，但均可能丢失关键局部细节或全局上下文。本文提出PerLA，一种更具感知力的3D语言助手，能同时捕捉高分辨率局部细节和低分辨率整体上下文。该方法利用希尔伯特曲线保持点云局部性，并通过交叉注意力与图神经网络实现局部到全局信息的有效聚合。此外，我们设计了一种新的局部表征一致性损失，提升训练稳定性。PerLA在多项基准上超越现有3D语言助手：在ScanQA上问答性能提升最高达+1.34 CiDEr；在ScanRefer和Nr3D上密集描述任务分别提升+4.22和+3.88。
原文摘要 · Abstract (English)
Enabling Large Language Models (LLMs) to understand the 3D physical world is an emerging yet challenging research direction. Current strategies for processing point clouds typically downsample the scene or divide it into smaller parts for separate analysis. However, both approaches risk losing key local details or global contextual information. In this paper, we introduce PerLA, a 3D language assistant designed to be more perceptive to both details and context, making visual representations more informative for the LLM. PerLA captures high-resolution (local) details in parallel from different point cloud areas and integrates them with (global) context obtained from a lower-resolution whole point cloud. We present a novel algorithm that preserves point cloud locality through the Hilbert curve and effectively aggregates local-to-global information via cross-attention and a graph neural network. Lastly, we introduce a novel loss for local representation consensus to promote training stability. PerLA outperforms state-of-the-art 3D language assistants, with gains of up to +1.34 CiDEr on ScanQA for question answering, and +4.22 on ScanRefer and +3.88 on Nr3D for dense captioning. https://gfmei.github.io/PerLA/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。