arXiv:2605.30087cs.AI2026-05被引 1

测试智能体在多源矛盾记忆中精准问答的能力,提供可复现的评估基准。

Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison

论文配图:Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison
图 1 · 摘自论文原文
  • 构建包含34,560个实例的对抗性记忆问答数据集,模拟真实冲突场景。
  • 融合方法最高达80.3%准确率,带拒答机制时选择性准确率达85.3%。
  • 揭示不同模型在复杂推理任务中的优劣,适合研究持续记忆与决策的团队。

个人人工智能代理正迈向持久化、多源记忆系统,带来新评估挑战:系统需在冲突或不完整证据中做出判断,不能依赖单一清晰历史。现有基准很少区分错误是源于输入证据还是方法本身的冲突处理。本文提出针对多源矛盾个人记忆的有选择性问答任务,要求系统基于冲突或不完整的来源作答,或在证据不足时拒绝回答。我们构建了一个基准,涵盖8类推理类型、480个角色、4个随机种子和18个问题模板,共34,560个实例,具备可控的源干扰和确定性真值。评估了无源基线、单源访问、结构化融合方法及前沿大模型的表现。最佳训练融合解析器达到80.3%准确率,最强仅提示大模型基线为70.0%。引入拒答后,同一解析器实现85.3%选择性准确率(78.3%覆盖率),最佳大模型达71.0%选择性准确率(95.4%覆盖率)。不同模型在各类推理中表现各异。所有数据、代码、缓存输出及生成流程均已开源。

原文摘要 · Abstract (English)

Emerging personal AI agents are moving toward persistent, multi-source memory. This creates an evaluation problem: systems must decide how to use conflicting or incomplete evidence; they cannot just retrieve facts from one clean history. Existing benchmarks rarely show whether an error came from the evidence given to a method or from the method's conflict-resolution step. We study this as selective QA over conflicting multi-source personal memory: systems answer based on conflicting, sometimes incomplete sources, or abstain when evidence is insufficient. We develop a benchmark containing 18 question templates across 8 reasoning types, 480 personas, 4 random seeds, and 34,560 instances, with controlled source distortions and deterministic ground truth. We evaluate the performance of baselines without access to any source, access to a single source, structured fusion methods, and frontier LLMs. The best trained fusion resolver reaches 80.3% accuracy, while the strongest prompt-only LLM baseline reaches 70.0%. With abstention, the same resolver reaches 85.3% selective accuracy at 78.3% coverage and the best LLM reaches 71.0% selective accuracy at 95.4% coverage. Different models have different strengths across reasoning types. We release the data, code, cached model outputs, and data-generating process for reuse.

个人记忆多源推理选择性问答大模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。