用贝叶斯模型自动生成真实可靠的记忆测试题，解决大模型助理记忆评估难题。
MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
- 基于贝叶斯关系网络与因果生成机制，减少大模型幻觉对问答的影响。
- 构建了日常场景下的评估数据集MemDaily，覆盖2000+用户消息与1万+问题。
- 提供可复现的基准测试，适合研究大模型记忆机制的学者使用。
基于大语言模型的智能助手广泛应用于个人助理场景，能够从用户消息中记忆信息并回答个性化问题。然而，由于难以根据用户消息可靠地构建问题与答案（QAs），其记忆能力缺乏客观、自动化的评估方法。本文提出MemSim，一种贝叶斯模拟器，可自动从生成的用户消息中构建兼具多样性与可扩展性的可靠QAs。我们引入贝叶斯关系网络（BRNet）和因果生成机制，有效缓解大模型幻觉对事实信息的影响，实现评估数据集的自动化构建。基于MemSim，我们生成了名为MemDaily的日常生活场景数据集，并进行了广泛的实验以验证方法有效性。同时，我们提供了针对不同记忆机制的基准评测体系。为促进学术交流，项目已开源至https://github.com/nuster1128/MemSim。
原文摘要 · Abstract (English)
LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset. To benefit the research community, we have released our project at https://github.com/nuster1128/MemSim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。