用记忆图结构减少大模型依赖，让强化学习更快学会复杂任务。
MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance
- 构建可演化的记忆图，融合高回报经验与大模型建议
- 仅需少量在线调用大模型，就达到接近持续监督的效果
- 适合需要快速学习但无法频繁调用大模型的场景
强化学习在稀疏或延迟奖励环境下常因样本效率低而表现不佳。大语言模型能提供子目标分解、合理轨迹和抽象先验，加速早期学习，但持续依赖其指导会带来可扩展性问题和信号不可靠风险。本文提出MIRA（记忆集成强化学习代理），通过结构化、动态演化的记忆图引导训练初期过程。该图存储决策相关的信息，包括轨迹片段与子目标结构，来源为高回报经验及大模型输出。此设计将大模型查询转化为持久记忆，避免实时依赖。基于记忆图生成效用信号，软性调节优势估计以影响策略更新，不改变原始奖励函数。随着训练推进，智能体策略逐渐超越初始大模型先验，效用项衰减，维持标准收敛性。理论分析表明，基于效用的奖励塑形能提升稀疏奖励环境下的早期学习效果。实验显示，MIRA优于常规强化学习基线，在返回值上媲美频繁调用大模型的方法，同时显著减少在线大模型调用次数。
原文摘要 · Abstract (English)
Reinforcement learning (RL) agents often suffer from high sample complexity in sparse or delayed reward settings due to limited prior structure. Large language models (LLMs) can provide subgoal decompositions, plausible trajectories, and abstract priors that facilitate early learning. However, heavy reliance on LLM supervision introduces scalability constraints and dependence on potentially unreliable signals. We propose MIRA (Memory-Integrated Reinforcement Learning Agent), which incorporates a structured, evolving memory graph to guide early training. The graph stores decision-relevant information, including trajectory segments and subgoal structures, and is constructed from both the agent's high-return experiences and LLM outputs. This design amortizes LLM queries into a persistent memory rather than requiring continuous real-time supervision. From this memory graph, we derive a utility signal that softly adjusts advantage estimation to influence policy updates without modifying the underlying reward function. As training progresses, the agent's policy gradually surpasses the initial LLM-derived priors, and the utility term decays, preserving standard convergence guarantees. We provide theoretical analysis showing that utility-based shaping improves early-stage learning in sparse-reward environments. Empirically, MIRA outperforms RL baselines and achieves returns comparable to approaches that rely on frequent LLM supervision, while requiring substantially fewer online LLM queries. Project webpage: https://narjesno.github.io/MIRA/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。