arXiv:2608.24189cs.CLcs.HC2026-08中稿 · EMNLP

现有记忆评估方法与用户满意度无关,新方法更贴近真实对话需求。

MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

论文配图:MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation
图 1 · 摘自论文原文
  • 用真实对话场景设计新评估方式,考察记忆的自然融入能力
  • 同一系统在直接问答中准确率78.8%,但在对话中仅7.9%被自然引用
  • 用户满意度与记忆自然融合度相关,而非传统问答准确率

对话式大模型的记忆系统传统上通过针对历史对话的直接事实提问(Direct QA)来评估:模型能否回忆起之前对话中的某个事实?我们在为期4个月的部署中测试了这一方法(40名用户,1,872次会话,7种记忆配置)。现有基准的Direct QA准确率在19.7%至70.1%之间波动,但用户满意度未受影响。我们推测,现有基准衡量的是被触发的检索能力(被问时才回忆),而真实对话需要的是自然整合能力(自主识别相关性并自然融入回应)。为此,我们提出MemUse,从实际部署中提取真实用户引导的记忆时刻,采用关注整合性的判断标准评估回应的自然程度。在固定模型和上下文的情况下,同一系统在Direct QA中得分为78.8%,但在对话中仅7.9%的事实被自然引用,差距达71个百分点。在这些时刻中,自然整合度与用户满意度相关,而Direct QA则无关。我们已将部署数据集与MemUse评估体系、所有标注及评分提示发布至https://github.com/ryuichi-sumida/memuse。

原文摘要 · Abstract (English)

Memory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a 4-month deployment (40 users, 1,872 sessions, 7 memory conditions). Existing-benchmark Direct QA varies from 19.7% to 70.1% across the 7 conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response). To examine this, we introduce MemUse, a set of real user-cued memory moments drawn from the deployment, scored by an integration-aware judgment of the natural conversational response. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation -- a 71-point gap. Within these moments, Natural Integration is associated with satisfaction, whereas Direct QA is not. We release the deployment corpus and MemUse together with all judgments and scoring prompts at https://github.com/ryuichi-sumida/memuse.

对话记忆评估方法用户满意度自然整合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。