为大模型智能体设计长时记忆评估基准,解决记忆能力在真实任务中表现差的问题。
MemGym: a Long-Horizon Memory Environment for LLM Agents

- 构建统一记忆推理接口,覆盖编码、搜索、工具使用等四类智能体场景。
- 提供独立记忆得分,剥离推理与工具使用干扰,精准评估记忆策略。
- 开发轻量级奖励模型MemRM,快速评估代码压缩质量,提升评测效率。
记忆是大模型智能体完成长时任务的核心能力。现有记忆评测主要聚焦多轮对话中的个性化信息保留,忽视了智能体执行过程中动态记忆的形成,导致其记忆系统在真实场景(如编程、网页导航)中迁移效果差。为此,我们提出MemGym,一个面向智能体记忆的统一基准,整合现有智能体环境与自研记忆驱动流程,涵盖五个评估赛道:工具使用对话(tau2-bench)、多轮深度研究搜索(MEMGYM-DR)、编码(SWE-Gym和MEMGYM-CODEQA)、计算机使用(WebArena-Infinity)。MemGym通过隔离记忆得分,将记忆性能与推理、检索及工具使用能力解耦,实现无混淆排序。其中,MEMGYM-CODEQA与MEMGYM-DR的合成流程长度可控、每阶段经消融验证,且与下游任务高度对齐。为使编码环境评测可学术化,我们训练了轻量级奖励模型MemRM(基于Qwen3-1.7B微调,采用QLoRA),以快速量化压缩质量,替代耗时的完整Docker运行。
原文摘要 · Abstract (English)
Memory is a central capability for LLM agents operating across long-horizon tasks. Existing memory benchmarks predominantly evaluate retention of personalized information in multi-turn chat scenarios, overlooking the dynamic memory formation that occurs during extended agent execution. Consequently, the memory systems they produce transfer poorly to realistic agentic environments, such as coding and web navigation. We present MemGym, a benchmark for agentic memory that unifies existing agent gyms and in-house memory-grounded pipelines behind one memory-reasoning interface. MemGym spans five evaluation tracks grouped into four agentic regimes: tool-use dialogue (tau2-bench), multi-turn deep-research search (MEMGYM-DR), coding (SWE-Gym and MEMGYM-CODEQA), and computer use (WebArena-Infinity). MemGym reports memory-isolated scores that decouple memory performance from reasoning, retrieval, and tool-use ability, so memory strategies can be ranked without those confounders. Our synthetic pipelines for MEMGYM-CODEQA and MEMGYM-DR are length-controllable, ablation-verified at every stage, and tightly aligned with downstream scenarios. To make evaluation on coding environments academically tractable, we train MemRM, a lightweight reward model (Qwen3-1.7B fine-tuned with QLoRA) that scores compression quality as a fast scalar read in place of full Docker rollouts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。