提出以记忆为中心的视觉问答框架,提升智能体探索效率。
Memory-Centric Embodied Question Answering
- 构建结构化记忆库,动态更新并按需检索
- 在多区域任务中实现9.9%性能提升
- 适合需要长期记忆的复杂环境导航任务
具身问答(EQA)要求智能体自主探索环境并回答上下文相关问题。传统框架包含规划、记忆、停止和回答四个模块,但现有方法仅在回答阶段使用记忆信息,导致探索冗余或不足。为此,本文提出以记忆为核心的MemoryEQA框架,建立记忆存储、更新与检索机制,使记忆信息贯穿整个探索过程。具体地,将观测转化为结构化文本存入固定结构的向量库;每步探索通过视点比较策略判断是否更新记忆;各模块执行前,采用基于熵的自适应检索策略获取最小且足够的记忆信息,并与当前观测融合输入对应模块。为评估记忆能力,构建基于HM3D的基准MT-HM3D,包含1,587个涉及多个目标的跨区域问答对。在HM-EQA、MT-HM3D和OpenEQA上的实验表明,该框架有效,尤其在MT-HM3D上相比基线模型提升9.9%,凸显记忆能力对解决复杂任务的关键作用。
原文摘要 · Abstract (English)
Embodied Question Answering (EQA) requires agents to autonomously explore and comprehend the environment to answer context-dependent questions. Typically, an EQA framework consists of four components: a planner, a memory module, a stopping module, and an answering module. However, the memory module is utilized inefficiently in existing methods, as the information it stores is leveraged solely for the answering module. Such a design may result in redundant or inadequate exploration, leading to a suboptimal success rate. To solve this problem, we propose MemoryEQA, an EQA framework centered on memory, which establishes mechanisms for memory storage, update, and retrieval, allowing memory information to contribute throughout the entire exploration process. Specifically, we convert the observation into structured textual representations, which are stored in a vector library following a fixed structure. At each exploration step, we utilize a viewpoint comparison strategy to determine whether the memory requires updating. Before executing each module, we employ an entropy-based adaptive retrieval strategy to obtain the minimal yet sufficient memory information that satisfies the requirements of different modules. The retrieved module-specific information is then integrated with the current observation as input to the corresponding module. To evaluate EQA models' memory capabilities, we constructed the benchmark based on HM3D called MT-HM3D, comprising 1,587 question-answer pairs involving multiple targets across various regions, which requires agents to maintain memory of exploration-acquired target information. Experimental results on HM-EQA, MT-HM3D, and OpenEQA demonstrate the effectiveness of our framework, where a 9.9% performance gain on MT-HM3D compared to baseline models further underscores the memory capability's pivotal role in solving complex tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。