让大模型记忆系统精准处理图文混合问题,提升长时多模态对话能力。
V-Mem: Modality-Routed Retrieval for Long-Term Multimodal Agentic Memory

- 按查询模态路由检索,避免跨模态误匹配
- 用LLM生成锚点提高相关性,图文联合提问准确率达0.87
- 适合需要长期多模态交互的智能代理应用
用户与大模型代理的交互日益多模态:对话中文本与图像交织,后续问题可能针对任一模态。然而多数代理记忆系统仍以文本为中心,即使支持多模态对话,对视觉相关问题仍表现不佳。我们发现根源在于其依赖的相似性搜索假设:在索引空间中,查询应靠近能回答它的相关证据。但在多模态场景下,这一假设被两个缺口破坏:模态间隙导致查询更接近同模态内容而非跨模态证据;相似性-相关性间隙则表现为最相似的内容往往并非答案,尤其当查询同时含文本和图像时,答案需融合两者信息。为此,我们提出V-Mem,一种多模态代理记忆系统,通过仅从查询中识别查询与目标证据的模态,实现检索路由。为克服模态间隙,V-Mem将对话分轮组织,仅在同一轮内返回目标模态内容,避免跨模态比较。为弥合相似性-相关性间隙,它使用由大模型生成的锚点进行搜索:对仅需图像的纯文本查询,生成假设描述作为锚点;对需图文结合才能回答的查询,采用查询文本+从图像中提取的关键字组合构成增强锚点。在Mem-Gallery上,V-Mem获得0.82的LLM评分,优于第二好的基线(0.56),尤其在含图像的问题上差距最大(0.87,无基线超0.47);在LoCoMo上得分为0.69,优于基线0.58。
原文摘要 · Abstract (English)
Interaction between users and LLM agents is increasingly multimodal: conversations interleave text with images, and a later question may target either. Yet most agent memories are designed around text, and even the few that support multimodal conversations still fail on vision-related questions. We trace this failure to an assumption behind the similarity search they rely on: in the index space, a query lies close to the relevant evidence that answers it. In multimodal settings, two gaps break it. By the modality gap, a query lies closer to memory content of its own modality than to evidence in another, even in a trained joint embedding space. By the similarity-relevance gap, the content most similar to a query is often not the evidence that answers it, most acutely when a query carries both text and image and its evidence resembles neither part alone. We present V-Mem, a multimodal agentic memory system that routes retrieval by the modality of the query and that of the target evidence, both recognized from the query alone. To cross the modality gap, V-Mem organizes the conversation into rounds and returns the target-modality content from the same round as the match, without comparing across modalities. To close the similarity-relevance gap, it searches with an LLM-generated anchor that sits closer to the relevant evidence than the query does: a hypothetical caption for a text-only query seeking an image, and an enriched search anchor, the query text plus relevant keywords extracted from the query image, when the evidence is reachable only by combining the two. On Mem-Gallery, V-Mem reaches an LLM-judge score of 0.82 versus 0.56 for the second best, with the largest margin on questions carrying an image (0.87, no baseline above 0.47); on LoCoMo it scores 0.69 versus 0.58.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。