用细粒度归因反馈训练大模型记忆策略,提升长期对话问答能力。
AttriMem: Attribution-Guided Process Feedback for Agent Memory Construction

- 基于令牌级贡献生成中间奖励,解决记忆构建中的信用分配难题。
- 在长时对话问答任务中超越检索、启发式和传统强化学习基线。
- 适合需要稳定记忆优化的复杂推理类Agent研发者参考。
有效记忆对大模型智能体至关重要,但其构建仍具挑战。记忆构建策略决定信息的提取、存储、更新、压缩或丢弃。启发式方法依赖主观、任务特定规则,易与下游目标错位,限制跨任务适应性;而基于强化学习的方法虽利用任务反馈,但多采用结果或模块级奖励,信号粗糙,无法识别哪些中间记忆内容支持最终答案,造成细粒度信用分配瓶颈。然而,构造此类过程反馈极难,因中间记忆决策无唯一真实目标,且适当信用随智能体不确定推理轨迹变化,难以预先设定。本文提出AttriMem,一种基于归因引导的过程反馈框架,用于通过强化学习学习记忆构建策略。AttriMem在全局结果奖励基础上,引入由令牌级贡献推导出的局部奖励。长时对话问答实验表明,AttriMem优于基于检索、启发式及强化学习的基线方法,具备跨基准和答案模型的泛化能力,并稳定了强化学习优化过程。
原文摘要 · Abstract (English)
Effective memory is crucial for LLM agents, yet constructing it effectively remains challenging. A memory-construction policy decides what information to extract, store, update, compress, or discard as interactions accumulate. Heuristic memory methods rely on subjective, task-specific rules, which can misalign with downstream objectives and limit cross-task adaptability. RL-based methods, by contrast, learn from task feedback but mainly use outcome- or module-level rewards. These coarse signals indicate task success but cannot identify which intermediate memory contents support the final answer, creating a fine-grained credit-assignment bottleneck. However, constructing such process feedback is prohibitively difficult because intermediate memory decisions lack unique ground-truth targets, while the appropriate credit varies with the agent's uncertain reasoning trajectory and therefore cannot be specified in advance. We propose AttriMem, an attribution-guided process-feedback framework for learning memory-construction policies with RL. AttriMem augments the global outcome reward with local rewards derived from token-level contributions to the final answer. Experiments on long-horizon dialogue question answering show that AttriMem outperforms retrieval-based, heuristic, and RL-based baselines, generalizes across benchmarks and answer models, stabilizes RL optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。