构建首个同时评估对话记忆与长文档推理的基准数据集。
MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning
- 设计50个微型世界,融合多轮对话与长文档,生成1000个问答对。
- 75.1%的问题需先查对话历史再找文档答案,体现联合推理挑战。
- 发现现有模型在联合任务上表现显著下降,凸显新架构需求。
AI系统亟需同时处理多轮对话历史与长文档深度理解,但现有基准未兼顾二者。我们提出MemoryDocDataSet,一个包含50个微型世界、1000个问答对的合成数据集,每条实例包含3-5个角色、跨月时间线事件图、3-5篇长文档(20,000–50,000词元,来自Caselaw Access Project)、基于文档的多轮对话,以及20个涵盖五类推理的问答对。核心特征是‘混合来源标签’:问题需先通过对话历史定位相关文档,再从中提取答案,此类问题占75.1%。通过大模型作为裁判的提示敏感性自一致性分析,数据集质量经验证,50个微型世界中位数Cohen's κ为0.634。我们评估了六种基线配置,包括截断上下文、长上下文LLM、检索增强生成(RAG)和记忆系统。最佳基线(RAG-Both)整体F1为0.358,混合问题得分为0.342;仅文档检索(RAG-Doc)在混合问题上仅0.267,尽管文档独立试题达0.453,揭示明显的联合检索差距,推动统一对话记忆与长文档导航的架构发展。数据集、生成管道及所有基线代码均已开源。
原文摘要 · Abstract (English)
AI systems increasingly need to combine two demanding capabilities: navigating multi-session conversation history and performing deep reading comprehension within long documents. Yet no existing benchmark evaluates both simultaneously. We introduce MemoryDocDataSet, a synthetic benchmark of 50 micro-worlds and 1,000 QA pairs in which each instance comprises 3-5 personas, a temporal event graph spanning months of activity, 3-5 real long documents (20,000-50,000 tokens each sourced from the Caselaw Access Project), multi-session conversations grounded on those documents, and 20 question-answer pairs across five reasoning categories. The defining feature is the Hybrid source tag: questions requiring a system to first navigate conversation history to identify which document is relevant, then extract the answer from within that document. Hybrid questions account for 75.1% of the dataset. Dataset quality is characterised through a prompt-sensitivity self-consistency analysis using LLM-as-judge, yielding a median Cohen's $κ= 0.634$ across all 50 micro-worlds. We evaluate six baseline configurations spanning truncated context, long-context LLMs, retrieval-augmented generation (RAG), and memory systems. The best baseline (RAG-Both) achieves 0.358 overall F1 and 0.342 on Hybrid. Document-only retrieval (RAG-Doc) collapses to 0.267 on Hybrid despite achieving 0.453 on Doc-only questions, demonstrating a clear joint-retrieval gap that motivates architectures unifying conversational memory with long-document navigation. We release the dataset, generation pipeline, and all baseline implementations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。