让大模型通过回顾过往经验，更聪明地做决策。
Retrospex: Language Agent Meets Offline Reinforcement Learning Critic
- 用离线强化学习回顾历史经验，生成动作价值。
- 结合大模型推理与经验价值，提升任务表现。
- 适合需要长期交互的复杂任务场景。
大型语言模型（LLMs）具备丰富的知识和常识推理能力，是构建强大智能体的重要基础。然而，现有基于大模型的智能体框架未能充分挖掘历史经验以优化自身。本文提出一种名为Retrospex的新框架，通过深度分析过往经验来解决这一问题。不同于直接将经验注入大模型上下文的方法，Retrospex将大模型的动作概率与一个基于离线经验训练的强化学习（RL）评价器估算的动作价值相结合。此外，Retrospex引入动态动作重评分机制，使依赖环境交互的任务更重视经验驱动的价值。我们在ScienceWorld、ALFWorld和Webshop环境中评估了Retrospex，结果表明其优于多个先进的基线方法。
原文摘要 · Abstract (English)
Large Language Models (LLMs) possess extensive knowledge and commonsense reasoning capabilities, making them valuable for creating powerful agents. However, existing LLM agent frameworks have not fully utilized past experiences for improvement. This work introduces a new LLM-based agent framework called Retrospex, which addresses this challenge by analyzing past experiences in depth. Unlike previous approaches, Retrospex does not directly integrate experiences into the LLM's context. Instead, it combines the LLM's action likelihood with action values estimated by a Reinforcement Learning (RL) Critic, which is trained on past experiences through an offline ''retrospection'' process. Additionally, Retrospex employs a dynamic action rescoring mechanism that increases the importance of experience-based values for tasks that require more interaction with the environment. We evaluate Retrospex in ScienceWorld, ALFWorld and Webshop environments, demonstrating its advantages over strong, contemporary baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。