让智能体在问答时动态更新3D场景图，提升理解精度
GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering
- 通过API调用实时修改场景图、导航日志和任务笔记
- 在OpenEQA上比纯图像基线高3.0%，仅需1/5输入帧
- 无需训练或数据收集，适合需要灵活感知的机器人应用
结构化场景表征是具身智能体的核心，能将原始感官流整合为可读、模块化且可搜索的形式。由于计算开销大，许多方法在任务前预先构建此类表征。然而当任务需求变化时，静态方法可能遗漏关键物体、空间关系和细节。我们提出GraphPad，一种可通过API调用动态调整的可变结构记忆：包含可修改的场景图、按帧索引内容的导航日志，以及用于任务笔记的临时草稿区。三者共同构成一个动态工作区，始终保持完整、实时且与智能体对场景及任务的理解一致。在OpenEQA基准上，GraphPad达到55.3%准确率，较使用相同视觉-语言模型的纯图像基线提升3.0%，同时仅需五分之一的输入帧。结果表明，在推理时通过语言驱动方式优化三维记忆，可在不增加训练或数据采集的前提下生成更丰富的表征。
原文摘要 · Abstract (English)
Structured scene representations are a core component of embodied agents, helping to consolidate raw sensory streams into readable, modular, and searchable formats. Due to their high computational overhead, many approaches build such representations in advance of the task. However, when the task specifications change, such static approaches become inadequate as they may miss key objects, spatial relations, and details. We introduce GraphPad, a modifiable structured memory that an agent can tailor to the needs of the task through API calls. It comprises a mutable scene graph representing the environment, a navigation log indexing frame-by-frame content, and a scratchpad for task-specific notes. Together, GraphPad serves as a dynamic workspace that remains complete, current, and aligned with the agent's immediate understanding of the scene and its task. On the OpenEQA benchmark, GraphPad attains 55.3%, a +3.0% increase over an image-only baseline using the same vision-language model, while operating with five times fewer input frames. These results show that allowing online, language-driven refinement of 3-D memory yields more informative representations without extra training or data collection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。