为智能助手设计可落地的多模态记忆系统，提升真实世界信息理解能力。
A Grounded Memory System For Smart Personal Assistants
- 融合视觉语言模型与大模型，实现感知中信息一致提取。
- 知识图谱结合向量嵌入，高效管理实体间关系。
- 语义搜索+图查询生成，支持基于记忆的问答生成。
众多代理型AI应用——从阿尔茨海默病患者的认知助手到机器人——都需要一个扎根于现实的鲁棒记忆系统。本文提出一个由三部分组成的记忆系统：首先，利用视觉语言模型进行图像描述和实体消歧，并结合大语言模型实现感知过程中的信息一致性提取；其次，将提取的信息以增强向量嵌入的知识图谱形式存储，高效管理关系信息；第三，通过语义搜索与图查询生成相结合，实现基于检索增强生成的问答。我们通过一个真实世界案例展示了该系统的运作方式与潜力。
原文摘要 · Abstract (English)
A wide variety of agentic AI applications - ranging from cognitive assistants for dementia patients to robotics - demand a robust memory system grounded in reality. In this paper, we propose such a memory system consisting of three components. First, we combine Vision Language Models for image captioning and entity disambiguation with Large Language Models for consistent information extraction during perception. Second, the extracted information is represented in a memory consisting of a knowledge graph enhanced by vector embeddings to efficiently manage relational information. Third, we combine semantic search and graph query generation for question answering via Retrieval Augmented Generation. We illustrate the system's working and potential using a real-world example.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。