让大模型同时理解环境点云和视角视频，提升真实世界推理能力
Spatio-Temporal LLM: Reasoning about Environments and Actions
- 融合点云、视频与文本输入，构建时空联合理解框架
- 在REA数据集上显著超越现有模型，空间与时间理解更准确
- 适合需要环境感知与动作推理的机器人、自动驾驶场景
尽管多模态大语言模型（MLLM）取得显著进展，但面对同时涉及完整环境点云与局部视角视频的动作提示时仍表现不佳。为解决这一问题，我们构建了大规模「环境与行为推理」（REA）数据集，发现现有模型在时空提示下表现有限。基于此，提出两种时空大模型（STLLM）基线：1）直接融合点云、视频与文本表示；2）在解码前对空间上下文、视频与文本进行对齐。两者均增强对环境的空间理解与视频的时间定位能力。在REA数据集上，新方法显著优于现有模型，验证设计有效性。代码与数据已公开。
原文摘要 · Abstract (English)
Despite significant recent progress of Multimodal Large Language Models (MLLMs), current MLLMs are challenged by "spatio-temporal" prompts, i.e., prompts that refer to 1) the entirety of an environment encoded in a point cloud that the MLLM should consider; and simultaneously also refer to 2) actions that happened in part of the environment and are encoded in a short ego-centric video clip. However, such a holistic spatio-temporal understanding is important for agents operating in the real world. To address this challenge, we first develop a framework to collect a large-scale dataset. Using the collected "Reasoning about Environments and Actions" (REA) dataset, we show that recent MLLMs indeed struggle to correctly answer "spatio-temporal" prompts. Building on this dataset, we study two spatio-temporal LLM (STLLM) baselines: 1) STLLM-3D, which directly fuses point cloud, video, and text representations as inputs to the LLM; and 2) STLLM-Aligner, which aligns spatial context with video and text before LLM decoding. Both baselines aim to enhance spatial understanding of environments and temporal grounding of egocentric observations. On REA, the STLLM baselines outperform existing models, demonstrating the effectiveness of our designs. Code and data are available at https://zoezheng126.github.io/STLLM-website/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。