用低维状态压缩记忆反馈，解决大模型记忆中的奖励污染问题。
RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States
- 用固定维度的极性与动态因子分解记忆状态，压缩反馈空间。
- 反馈密度提升6倍，内存大小减少84.4%，冷启动率降低80%。
- 适合长期自进化智能体，尤其在记忆效率与稳定性要求高的场景。
基于学习的记忆系统在自进化大模型智能体中面临两个紧密耦合的挑战：其一，轨迹索引的效用随交互历史增长，导致有限反馈分散于不断扩大的状态空间；其二，因轨迹级奖励联合分配给共检索记忆，无关经验可能获得误导性效用更新，陷入记忆-奖励陷阱。为此，我们提出低阶记忆强化学习（RoMeRL），通过结果极性和记忆动态因子分解，以固定维度的每任务记忆状态表示不断增长的轨迹效用空间。新经验通过一组固定语义坐标纳入，内容随时间更新或替换，从而将反馈集中于有界效用支持域。理论上，该低阶参数化提升了每个效用坐标的平均反馈量，并刻画了通用坐标转移模型下的错误坐标准稳占用。实证上，在ALFWorld和LifelongAgentBench上，RoMeRL显著提升任务性能，冷启动比（Cold-Q ratio）下降80.0%，反馈密度提升约6.0倍，维持内存大小减少84.4%，大模型调用次数减少21.1%。结果表明，低阶效用状态能高效支持自进化记忆，同时限制持续奖励污染。代码已开源：https://github.com/YOUNG-fnxm/RoMeRL
原文摘要 · Abstract (English)
Learning-based memory systems for self-evolving LLM agents face two tightly coupled challenges. First, trajectory-indexed utilities grow with the interaction history, thereby dispersing limited feedback over an ever-expanding state space. Second, because trajectory-level rewards are jointly assigned to co-retrieved memories, irrelevant experiences may receive misleading utility updates and consequently enter the memory-reward trap. To address these challenges, we introduce Reduced-Order Memory Reinforcement Learning (RoMeRL), which represents the growing trajectory-indexed utility space using a fixed-dimensional per-task memory state factorized by outcome polarity and memory dynamics. RoMeRL incorporates new experiences through a fixed set of semantic coordinates whose contents are updated or replaced over time, thereby concentrating feedback over a bounded utility support. Theoretically, we show that this reduced-order parameterization increases the average feedback received by each utility coordinate and characterize the steady-state occupancy of erroneous coordinates under a generic coordinate-transition model. Empirically, across ALFWorld and LifelongAgentBench, RoMeRL improves task performance, reduces the Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces the maintained memory size by 84.4%, and cuts LLM calls by 21.1%. These results show that reduced-order utility states support efficient self-evolving agent memory while limiting persistent reward contamination. Code is available at: https://github.com/YOUNG-fnxm/RoMeRL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。