提出新型非参数记忆框架，提升智能体长期探索与问答能力。
HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answering
- 分立情景与语义记忆，先检索后视觉验证
- 在A-EQA上提升7.3%（LLM-Match）和11.4%（LLM MatchXSPL）
- 适合需要长期记忆与跨环境推理的智能体研究
将多模态大语言模型作为具身智能体的‘大脑’仍面临长期观察与有限上下文预算的挑战。现有记忆方法依赖文本摘要，丢失丰富的视觉与空间细节，在非平稳环境中表现脆弱。本文提出一种非参数化记忆框架，显式分离情景记忆与语义记忆。采用‘检索优先、推理辅助’范式，通过语义相似性召回情景经验，并经由视觉推理验证，实现无需严格几何对齐的过往观测复用。同时引入程序化规则提取机制，将经验转化为结构化可重用的语义记忆，增强跨环境泛化能力。大量实验表明，在具身问答与探索基准上达到领先性能：A-EQA上LLM-Match提升7.3%，LLM MatchXSPL提升11.4%；GOAT-Bench上成功率提升7.7%，SPL提升6.8%。分析显示，情景记忆主要提升探索效率，语义记忆强化复杂推理能力。
原文摘要 · Abstract (English)
Deploying Multimodal Large Language Models as the brain of embodied agents remains challenging, particularly under long-horizon observations and limited context budgets. Existing memory assisted methods often rely on textual summaries, which discard rich visual and spatial details and remain brittle in non-stationary environments. In this work, we propose a non-parametric memory framework that explicitly disentangles episodic and semantic memory for embodied exploration and question answering. Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning, enabling robust reuse of past observations without rigid geometric alignment. In parallel, we introduce a program-style rule extraction mechanism that converts experiences into structured, reusable semantic memory, facilitating cross-environment generalization. Extensive experiments demonstrate state-of-the-art performance on embodied question answering and exploration benchmarks, yielding a 7.3% gain in LLM-Match and an 11.4% gain in LLM MatchXSPL on A-EQA, as well as +7.7% success rate and +6.8% SPL on GOAT-Bench. Analyses reveal that our episodic memory primarily improves exploration efficiency, while semantic memory strengthens complex reasoning of embodied agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。