构建首个评估大模型持续学习能力的综合性基准，模拟真实用户反馈场景。
MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems
- 设计用户反馈模拟框架，覆盖多领域多语言任务
- 现有主流方法在持续学习上效果不佳，效率远未达标
- 适合研究大模型记忆与持续优化的学者使用
扩大数据、参数和推理时计算量是提升大模型系统（LLMsys）性能的主流方法，但其上限已接近，因高质量数据逐渐耗尽，且更大算力带来的增益日益微弱。受人类及传统AI系统通过实践学习能力启发，为大模型系统构建记忆与持续学习框架成为近年研究热点。然而，现有评测基准多聚焦于同质化长文本阅读理解任务，未能有效检验模型在服务过程中从累积用户反馈中学习的能力。为此，我们提出一种用户反馈模拟框架，并构建了一个涵盖多领域、多语言、多类型任务的综合性基准，用于评估大模型系统的持续学习能力。实验表明，当前先进基线方法在有效性与效率方面均未达到理想水平。本基准旨在推动未来大模型记忆机制与优化算法的研究。官网：https://memorybench.thuir.cn 代码：https://github.com/THUIR/MemoryBench 数据集：https://huggingface.co/datasets/THUIR/MemoryBench Full数据集：https://huggingface.co/datasets/THUIR/MemoryBench-Full
原文摘要 · Abstract (English)
Scaling up data, parameters, and test-time computation has been the mainstream methods to improve LLM systems (LLMsys), but their upper bounds are almost reached due to the gradual depletion of high-quality data and marginal gains obtained from larger computational resource consumption. Inspired by the abilities of human and traditional AI systems in learning from practice, constructing memory and continual learning frameworks for LLMsys has become an important and popular research direction in recent literature. Yet, existing benchmarks for LLM memory often focus on evaluating the system on homogeneous reading comprehension tasks with long-form inputs rather than testing their abilities to learn from accumulated user feedback in service time. Therefore, we propose a user feedback simulation framework and a comprehensive benchmark covering multiple domains, languages, and types of tasks to evaluate the continual learning abilities of LLMsys. Experiments show that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying, and we hope this benchmark could pave the way for future studies on LLM memory and optimization algorithms. Website: https://memorybench.thuir.cn Code: https://github.com/THUIR/MemoryBench Data: https://huggingface.co/datasets/THUIR/MemoryBench Data-Full: https://huggingface.co/datasets/THUIR/MemoryBench-Full
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。