通过评估记忆中的信念不确定性，提升大模型长任务推理的准确性。
Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
- 用信念熵作为代理指标，实时检测记忆中信息丢失问题
- 在175万词上下文下仍保持97.1%性能，显著优于现有方法
- 适合需要长期规划与高可靠性推理的应用场景
记忆增强型大模型代理通过递归总结交互轨迹来处理复杂长时序任务。然而，现有方法通常使用基于结果的强化学习训练记忆策略，无法定位中间记忆质量下降的位置。随着交互进行，模糊的递归摘要逐步丢弃任务相关信 息并引入语义噪声，加剧信念偏差，干扰代理对潜在任务状态的估计，最终导致长时序推理失败。为此，我们提出信念熵（Belief Entropy），一种自监督代理，用于探测当前记忆下模型对潜在任务状态的不确定程度。基于此，我们提出元认知记忆策略优化（MMPO）。不同于仅依赖稀疏结果信号，MMPO通过显式惩罚引发高认知不确定性的摘要，提供细粒度、针对记忆的监督。实验表明，MMPO在多种长时序任务上持续优于现有方法，在175万词上下文规模下仍保持97.1%的性能。
原文摘要 · Abstract (English)
Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality degrades. As interactions unfold, ambiguous recursive summaries progressively discard task-relevant information and introduce semantic noise. This exacerbates belief deviation, obscuring the agent's estimate of the latent task state and ultimately derailing long-horizon reasoning. We therefore argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries. To this end, we introduce Belief Entropy, a self-supervised proxy that probes how uncertain the model remains about the latent task state given its current memory. Based on this proxy, we propose Metacognitive Memory Policy Optimization (MMPO). Instead of relying only on sparse outcome-based signals, MMPO provides fine-grained, memory-specific supervision via explicitly penalizing summaries that induce high epistemic uncertainty. Experiments show that MMPO consistently outperforms existing methods on diverse long-horizon tasks, maintaining 97.1% performance even when scaled to 1.75M-token contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。