让智能体像人一样记住图文信息，提升任务效率与准确性。
AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory
- 用语义标签构建图结构记忆库，支持跨模态概念检索。
- 在ImageNet分类上比传统方法快3.5倍，准确率更高。
- 适合需要长期记忆和多模态理解的智能体系统研究者。
基于大模型检索增强生成（RAG）的成功，学界开始关注为智能体系统引入外部记忆数据库。然而现有系统仅存储文本信息，忽视了多模态信号的重要性。受认知科学中人类记忆多模态特性的启发，我们提出AUGUSTUS，一种符合人类记忆机制的多模态智能体系统。该系统包含四个循环阶段：(i) 编码：理解输入；(ii) 存储：保存关键信息；(iii) 检索：从记忆中搜索相关上下文；(iv) 执行：完成任务。不同于传统向量数据库，我们提出将信息概念化为语义标签，并将其与上下文关联，存入图结构的多模态上下文记忆库中，实现高效的概念驱动检索。实验表明，本系统在图像分类任务中较传统多模态RAG方法更快3.5倍，且在MSC基准测试中优于MemGPT。
原文摘要 · Abstract (English)
Riding on the success of LLMs with retrieval-augmented generation (RAG), there has been a growing interest in augmenting agent systems with external memory databases. However, the existing systems focus on storing text information in their memory, ignoring the importance of multimodal signals. Motivated by the multimodal nature of human memory, we present AUGUSTUS, a multimodal agent system aligned with the ideas of human memory in cognitive science. Technically, our system consists of 4 stages connected in a loop: (i) encode: understanding the inputs; (ii) store in memory: saving important information; (iii) retrieve: searching for relevant context from memory; and (iv) act: perform the task. Unlike existing systems that use vector databases, we propose conceptualizing information into semantic tags and associating the tags with their context to store them in a graph-structured multimodal contextual memory for efficient concept-driven retrieval. Our system outperforms the traditional multimodal RAG approach while being 3.5 times faster for ImageNet classification and outperforming MemGPT on the MSC benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。