arXiv:2511.10523cs.CL2025-11被引 8

实验证明前150轮对话无需RAG,直接用长上下文更高效。

Convomem Benchmark: Why Your First 150 Conversations Don't Need RAG

  • 构建7.5万条问答的对话记忆基准,覆盖用户事实、偏好等多维度。
  • 前150轮对话中,全上下文方法准确率达70%-82%,远超RAG系统。
  • 适合研究对话记忆与低成本长上下文应用的开发者和研究人员。

我们提出一个全面的对话记忆评估基准,包含75,336个跨多样化类别(用户事实、助手回忆、回避、偏好、时间变化、隐含关联)的问题-答案对。现有基准虽推动领域发展,但受限于统计功效不足、数据生成不一致及评估灵活性差。本文分析对话记忆与检索增强生成(RAG)的关系:二者共享时间推理、隐式提取、知识更新和图表示等架构模式,但对话记忆从零开始逐步积累,使朴素方法在传统RAG中不可行,却在此场景下可行。结合近期长上下文有效性研究,发现全上下文方法在最复杂多消息证据任务中仍可达到70%-82%准确率,而基于RAG的记忆系统如Mem0在对话历史少于150轮时仅达30%-45%。分析揭示实用转折点:长上下文在前30轮表现优异,至150轮仍具可行性,超过后因成本与延迟问题需转向混合或RAG方案。这表明对话记忆的小语料优势——可实现完整搜索与重排序——应获得独立研究关注,而非简单套用通用RAG解决方案于对话历史。

原文摘要 · Abstract (English)

We introduce a comprehensive benchmark for conversational memory evaluation containing 75,336 question-answer pairs across diverse categories including user facts, assistant recall, abstention, preferences, temporal changes, and implicit connections. While existing benchmarks have advanced the field, our work addresses fundamental challenges in statistical power, data generation consistency, and evaluation flexibility that limit current memory evaluation frameworks. We examine the relationship between conversational memory and retrieval-augmented generation (RAG). While these systems share fundamental architectural patterns--temporal reasoning, implicit extraction, knowledge updates, and graph representations--memory systems have a unique characteristic: they start from zero and grow progressively with each conversation. This characteristic enables naive approaches that would be impractical for traditional RAG. Consistent with recent findings on long context effectiveness, we observe that simple full-context approaches achieve 70-82% accuracy even on our most challenging multi-message evidence cases, while sophisticated RAG-based memory systems like Mem0 achieve only 30-45% when operating on conversation histories under 150 interactions. Our analysis reveals practical transition points: long context excels for the first 30 conversations, remains viable with manageable trade-offs up to 150 conversations, and typically requires hybrid or RAG approaches beyond that point as costs and latencies become prohibitive. These patterns indicate that the small-corpus advantage of conversational memory--where exhaustive search and complete reranking are feasible--deserves dedicated research attention rather than simply applying general RAG solutions to conversation histories.

对话记忆RAG长上下文评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。