让大模型记忆操作更精准，提升长任务完成率
Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management
- 用分块问答任务提供每步即时反馈，避免奖励稀疏
- 根据记忆项在推理中的作用重分配奖励，精准定位有效操作
- 适用于多种模型架构，尤其适合复杂长流程任务
有效的记忆管理对大型语言模型代理完成长周期任务至关重要。现有方法依赖最终任务表现作为主要奖励，导致奖励稀疏且信用分配无效，难以指导具体记忆操作。为此，我们提出 Fine-Mem，一种统一的细粒度反馈对齐框架。首先引入分块级步骤奖励，通过辅助的分块特定问答任务提供即时步骤级监督；其次设计证据锚定奖励归因机制，基于推理中使用的具体记忆项重新分配全局奖励。二者协同实现稳定策略优化，并使局部记忆操作与长期记忆效用对齐。在 Memalpha 与 MemoryAgentBench 上的实验表明，Fine-Mem 持续优于强基线，在多个子任务上取得更高成功率。进一步分析显示其对不同模型配置和骨干网络具有良好的适应性与泛化能力。
原文摘要 · Abstract (English)
Effective memory management is essential for large language model agents to navigate long-horizon tasks. Recent research has explored using Reinforcement Learning to develop specialized memory manager agents. However, existing approaches rely on final task performance as the primary reward, which results in severe reward sparsity and ineffective credit assignment, providing insufficient guidance for individual memory operations. To this end, we propose Fine-Mem, a unified framework designed for fine-grained feedback alignment. First, we introduce a Chunk-level Step Reward to provide immediate step-level supervision via auxiliary chunk-specific question answering tasks. Second, we devise Evidence-Anchored Reward Attribution to redistribute global rewards by anchoring credit to key memory operations, based on the specific memory items utilized as evidence in reasoning. Together, these components enable stable policy optimization and align local memory operations with the long-term utility of memory. Experiments on Memalpha and MemoryAgentBench demonstrate that Fine-Mem consistently outperforms strong baselines, achieving superior success rates across various sub-tasks. Further analysis reveals its adaptability and strong generalization capabilities across diverse model configurations and backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。