用密集奖励强化大模型构建长期记忆，提升对话一致性。
MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards
- 通过生成会话级问题提供中间奖励，解决奖励稀疏问题
- 40亿参数模型超越闭源基线，长对话表现更优
- 适合研究长时对话、记忆增强系统的研究者
长时对话中保持一致性仍是大模型的核心挑战，传统检索机制难以捕捉历史状态的时序演变。尽管记忆增强框架提供了结构化替代方案，但现有系统依赖封闭模型的静态提示，或在稀疏奖励下训练效果不佳。本文提出MemBuilder，一种基于强化学习的框架，使模型能够协同构建多维度记忆，并采用属性化密集奖励。针对两大难题：(1) 轨迹级奖励稀疏性，通过合成会话级问题生成密集中间奖励；(2) 多维记忆可归因性，引入贡献感知梯度加权，按各组件对下游任务的影响动态调整策略更新。实验表明，该方法使40亿参数模型在长时对话基准上超越现有闭源基线，展现出强泛化能力。
原文摘要 · Abstract (English)
Maintaining consistency in long-term dialogues remains a fundamental challenge for LLMs, as standard retrieval mechanisms often fail to capture the temporal evolution of historical states. While memory-augmented frameworks offer a structured alternative, current systems rely on static prompting of closed-source models or suffer from ineffective training paradigms with sparse rewards. We introduce MemBuilder, a reinforcement learning framework that trains models to orchestrate multi-dimensional memory construction with attributed dense rewards. MemBuilder addresses two key challenges: (1) Sparse Trajectory-Level Rewards: we employ synthetic session-level question generation to provide dense intermediate rewards across extended trajectories; and (2) Multi-Dimensional Memory Attribution: we introduce contribution-aware gradient weighting that scales policy updates based on each component's downstream impact. Experimental results show that MemBuilder enables a 4B-parameter model to outperform state-of-the-art closed-source baselines, exhibiting strong generalization across long-term dialogue benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。