arXiv:2510.01483cs.ROcs.AI2025-10被引 1

用时空知识图谱让机器人记住场景,问答更快更可解释。

Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering

  • 从单目第一视角视频构建时空知识图谱,不依赖3D重建。
  • 查询时无需重看视频,延迟与视频长度无关,准确率媲美顶尖模型。
  • 适合需要长期记忆的机器人问答任务,支持真实机器人部署。

视觉语言模型(VLMs)在图像级场景理解上表现优异,但对长时第一视角视频的推理成本高:因缺乏持久记忆和显式空间表示,每次查询都需重新处理所有帧。我们提出VL-KnG,一种无需训练的框架,从单目第一视角视频构建时空知识图谱,融合细粒度场景图与全局拓扑图,无需3D重建。该框架分块处理视频,通过大语言模型实现时空对象关联(STOA)以维持物体身份持久性,并通过图增强检索(GER)结合子图检索与视觉定位回答问题。一旦构建完成,知识图谱可避免查询时重新访问原始视频,使查询延迟与视频长度解耦。在三个具身问答基准(OpenEQA、NaVQA及新提出的WalkieKnowledge)上,VL-KnG达到与前沿VLM相当的准确率,同时显著降低延迟,并提供可解释的图基推理;在多个设置中超越先前持久表示基线与开源VLM。在真实机器人上的部署验证了其实用性,随着观察历史增长,查询延迟保持稳定。作为持久、可查询的场景记忆,VL-KnG为具身智能体的记忆维护与知识更新提供了具体实现基础。

原文摘要 · Abstract (English)

Vision-language models (VLMs) demonstrate strong image-level scene understanding, but reasoning over long egocentric video remains costly: because VLMs maintain no persistent memory or explicit spatial representation, all sampled frames must be re-processed for every new query. We present VL-KnG, a training-free framework that constructs spatiotemporal knowledge graphs from monocular egocentric video, bridging fine-grained scene graphs and global topological graphs without 3D reconstruction. VL-KnG processes video in chunks, maintains persistent object identities via large language model (LLM)-based Spatiotemporal Object Association (STOA), and answers questions through Graph-Enhanced Retrieval (GER), which combines subgraph retrieval with visual grounding. Once constructed, the knowledge graph removes the need to revisit raw video at query time, decoupling query latency from video length. Across three embodied question answering benchmarks -- OpenEQA, NaVQA, and our newly introduced WalkieKnowledge -- VL-KnG achieves accuracy competitive with frontier VLMs while answering queries at substantially lower latency and with explainable, graph-grounded reasoning; it further surpasses prior persistent-representation baselines and open-weight VLMs in several settings. Deployment on a real robot demonstrates practical applicability, with query latency remaining stable as observation history grows. As a persistent, queryable scene memory, VL-KnG provides a concrete substrate for memory maintenance and knowledge updating in embodied AI agents.

具身智能知识图谱视觉问答持久记忆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。