让大模型智能体同时学习经验规则和策略，提升复杂交互任务表现
Joint Learning of Experiential Rules and Policies for Large Language Model Agents
- 从相同交互轨迹中联合优化规则池与策略模型
- 在阿尔福世界和网页购物任务上显著提升决策性能
- 适合需要长期经验积累的复杂交互场景
对于多步交互环境中的大语言模型智能体，如何有效利用积累的交互经验是一个关键挑战。现有方法通常将经验分为两类：一是以自然语言规则形式保存于模型外部用于提示；二是利用轨迹和反馈更新模型参数。前者易于理解但易与演进的策略脱节，后者虽能整体改进策略，但在稀疏奖励设置下对局部错误修正有限。本文提出联合学习经验规则与策略的JERP方法，从同一交互轨迹中同步更新长期经验规则池与策略模型。决策时，JERP检索任务相关的规则并结合历史交互进行条件化推理。每轮任务结束后，通过对比当前轨迹与参考成功轨迹，同时优化策略并修订规则池。该机制使规则池始终与策略保持一致，同时逐步将稳定有效的行为固化到模型中。在AlfWorld和WebShop上的实验表明，JERP在复杂交互任务中持续提升决策表现。
原文摘要 · Abstract (English)
For LLM agents in multi-step interactive environments, a key challenge is to make effective use of accumulated interaction experience. Existing work has typically separated two uses of such experience: keeping it outside the model as natural-language rules for later prompting, or using trajectories and feedback to update the model parameters. The former is easy to interpret but can fall out of sync with the evolving policy; the latter improves the policy more broadly but provides only limited correction for local mistakes in sparse-reward settings. We present Joint Learning of Experiential Rules and Policies for LLM Agents (JERP), which updates a long-term experiential-rule pool and the policy from the same interaction trajectories. At decision time, JERP retrieves task-relevant rules and conditions the agent on them together with the interaction history. After each episode, it uses the collected trajectories both to optimize the policy and to revise the rule pool by comparing current rollouts with reference successful trajectories. This coupling keeps the rule pool aligned with the evolving policy while allowing stable and effective behaviors to be gradually absorbed into the model itself. Experiments on AlfWorld and WebShop show that JERP yields consistent gains in decision performance for complex interactive tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。