arXiv:2412.11026cs.CVcs.AI2024-12被引 15

用大模型隐式理解动态场景,生成更准确的语义图谱。

SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation

  • 将视频转为语言信号,让大模型理解时空信息
  • 在Action Genome数据集上达到当前最优性能
  • 适合做自动驾驶与机器人感知系统的场景理解

动态场景包含复杂的时空信息,对移动机器人、无人机和自动驾驶系统做出决策至关重要。将这些场景解析为语义三元组<主体-谓词-客体>以实现精准的场景图生成(SGG)极具挑战性,原因在于其时空复杂度不断变化。受大语言模型(LLM)推理能力启发,我们提出SceneLLM,一种利用LLM作为强大场景分析器的新型框架,用于动态SGG。该框架引入视频到语言(V2L)映射模块,将视频帧转化为语言信号(场景标记),使输入更易于被LLM理解。为更好编码空间信息,我们设计了受汉字结构启发的空间信息聚合(SIA)方案,将空间数据编码为标记。通过最优传输(OT),从帧级标记序列生成捕捉视频时空信息的隐式语言信号。为进一步提升LLM处理此类隐式语言输入的能力,我们采用低秩适配(LoRA)进行微调。最后,使用基于Transformer的SGG预测器解码LLM的推理并预测语义三元组。该方法在Action Genome(AG)基准上取得当前最优结果,大量实验验证了SceneLLM在理解与生成准确动态场景图方面的有效性。

原文摘要 · Abstract (English)

Dynamic scenes contain intricate spatio-temporal information, crucial for mobile robots, UAVs, and autonomous driving systems to make informed decisions. Parsing these scenes into semantic triplets <Subject-Predicate-Object> for accurate Scene Graph Generation (SGG) is highly challenging due to the fluctuating spatio-temporal complexity. Inspired by the reasoning capabilities of Large Language Models (LLMs), we propose SceneLLM, a novel framework that leverages LLMs as powerful scene analyzers for dynamic SGG. Our framework introduces a Video-to-Language (V2L) mapping module that transforms video frames into linguistic signals (scene tokens), making the input more comprehensible for LLMs. To better encode spatial information, we devise a Spatial Information Aggregation (SIA) scheme, inspired by the structure of Chinese characters, which encodes spatial data into tokens. Using Optimal Transport (OT), we generate an implicit language signal from the frame-level token sequence that captures the video's spatio-temporal information. To further improve the LLM's ability to process this implicit linguistic input, we apply Low-Rank Adaptation (LoRA) to fine-tune the model. Finally, we use a transformer-based SGG predictor to decode the LLM's reasoning and predict semantic triplets. Our method achieves state-of-the-art results on the Action Genome (AG) benchmark, and extensive experiments show the effectiveness of SceneLLM in understanding and generating accurate dynamic scene graphs.

场景图生成大模型应用视频理解动态场景

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。