arXiv:2505.03581cs.CV2025-05被引 4

用动态图结构+大模型,让机器人理解人与物体的交互历史。

DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes

  • 将文本场景图序列编码为动态图,融合大模型推理能力。
  • 在STAR和AGQA数据集上比视觉方法高15-25%准确率。
  • 可处理图像输入,适合长时序机器人记忆任务。

动态环境中事件分析是智能体和机器人与人类互动的关键挑战。现有方法多依赖视觉模型,但通常从图像中隐式提取信息,缺乏可解释的空间-时间对象表示。为此,我们提出DyGEnc——一种动态图编码新方法,将压缩的时空结构观测表示与大语言模型的认知能力结合,实现基于文本场景图序列的高级问答。在STAR和AGQA数据集上的扩展评估表明,该方法在回答关于人-物交互历史的问题上,相比现有视觉方法提升15-25%。此外,通过基础模型提取显式文本场景图,该方法可无缝扩展至原始图像输入,机器人实验验证了其在轮式操作平台上的有效性。我们希望这些成果能推动基于压缩图结构的长时序机器人记忆实现。代码已开源:github.com/linukc/DyGEnc。

原文摘要 · Abstract (English)

The analysis of events in dynamic environments poses a fundamental challenge in the development of intelligent agents and robots capable of interacting with humans. Current approaches predominantly utilize visual models. However, these methods often capture information implicitly from images, lacking interpretable spatial-temporal object representations. To address this issue we introduce DyGEnc - a novel method for Encoding a Dynamic Graph. This method integrates compressed spatial-temporal structural observation representation with the cognitive capabilities of large language models. The purpose of this integration is to enable advanced question answering based on a sequence of textual scene graphs. Extended evaluations on the STAR and AGQA datasets indicate that DyGEnc outperforms existing visual methods by a large margin of 15-25% in addressing queries regarding the history of human-to-object interactions. Furthermore, the proposed method can be seamlessly extended to process raw input images utilizing foundational models for extracting explicit textual scene graphs, as substantiated by the results of a robotic experiment conducted with a wheeled manipulator platform. We hope that these findings will contribute to the implementation of robust and compressed graph-based robotic memory for long-horizon reasoning. Code is available at github.com/linukc/DyGEnc.

动态图视觉问答机器人记忆大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。