arXiv:2605.23067cs.CL2026-05

训练数据如何影响记忆增强型智能体?实验证明课程设计决定技能专精程度。

What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA

论文配图:What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA
图 1 · 摘自论文原文
  • 固定模型与算法,仅改变训练数据组合来研究课程影响
  • 混合课程在双基准测试中取得最高整体准确率(F1)
  • 单一外域训练可转移特定能力(如时间推理),适合针对性优化

强化学习(RL)已成为训练大语言模型智能体在多轮对话中操作外部记忆库的有效方法。现有工作仅在单一基准上训练,未揭示训练数据构成如何塑造智能体技能。本文开展受控实证研究,固定架构、强化学习算法与所有超参数,仅改变训练课程:领域内(LoCoMo)、跨基准混合(LoCoMo + LongMemEval)和纯外域(LongMemEval)。在两个基准与十类问题上,课程组成是技能专精的精细调控杠杆,而非性能的统一放大因子。混合课程在两评估集上均取得最强整体F1。仅在宽外域数据上训练虽整体表现弱,但能有效迁移时间推理能力。各类别差异远大于整体差异,表明单数值基准比较系统性低估了课程效应。此外,将GRPO适配单卡环境时发现:跨基准混合需过滤记忆库中的格式噪声以保留训练信号;在小组规模G=4下,二值精确匹配奖励无学习信号,推动采用连续奖励函数。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has emerged as a viable recipe for training LLM agents to reason over external memory banks in multi-session dialogue. Existing work trains exclusively on a single benchmark, leaving open how the composition of training data shapes the skills a memory agent acquires. We present a controlled empirical study that holds architecture, RL algorithm, and all hyperparameters fixed and varies only the training curriculum across three conditions: in-domain (LoCoMo), mixed-benchmark (LoCoMo + LongMemEval), and out-of-domain (LongMemEval only). Across two benchmarks and ten question types, curriculum composition acts as a fine-grained lever on specialization rather than a uniform scaling factor on performance. The mixed curriculum yields the strongest overall F1 on both evaluation sets. Training on a narrow out-of-domain set transfers a targeted skill - temporal reasoning - despite weak aggregate performance. Per-type differences substantially exceed aggregate differences, indicating that single-number benchmark comparisons systematically underreport curriculum effects. We further report two practical lessons from adapting GRPO to a single-GPU regime: cross-benchmark mixing requires filtering format-specific noise from memory banks to preserve training signal, and binary exact-match reward produces no learning signal at the small group sizes (G = 4) required on one GPU, motivating continuous reward functions in this regime.

强化学习记忆增强课程学习问答系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。