用SAC与事后重标注提升LLM智能体在线强化学习能力
SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling
- 将SAC与事后重标注引入LLM智能体在线学习，支持离线经验回放
- 在多目标环境中性能超越传统策略方法，提升策略效率
- 适合构建自主探索、自设目标的智能体，推动自驱型AI发展
近年来，大语言模型（LLMs）不仅作为生成模型，还被用于解决文本序列决策任务的智能体。当面对零样本能力不足的复杂环境时，近期研究显示在线强化学习（RL）可帮助LLM智能体交互式地发现并学习高效策略。然而，多数先前工作依赖于on-policy算法，严重限制了智能体可用的方法范围，如经验回放和事后重标注。这些方法对LLM学习智能体尤为重要，尤其是设计自主内在激励智能体时，该类智能体可自主采样并追求自身目标（即自驱智能体）。本文提出并研究了Soft Actor-Critic（SAC）与事后重标注在LLM智能体中的适应性应用。该方法不仅为自驱式在线学习的LLM智能体开辟路径，还能在经典多目标强化学习环境中优于on-policy方法。
原文摘要 · Abstract (English)
The past years have seen Large Language Models (LLMs) strive not only as generative models but also as agents solving textual sequential decision-making tasks. When facing complex environments where their zero-shot abilities are insufficient, recent work showed online Reinforcement Learning (RL) could be used for the LLM agent to discover and learn efficient strategies interactively. However, most prior work sticks to on-policy algorithms, which greatly reduces the scope of methods such agents could use for both exploration and exploitation, such as experience replay and hindsight relabeling. Yet, such methods may be key for LLM learning agents, and in particular when designing autonomous intrinsically motivated agents sampling and pursuing their own goals (i.e. autotelic agents). This paper presents and studies an adaptation of Soft Actor-Critic and hindsight relabeling to LLM agents. Our method not only paves the path towards autotelic LLM agents that learn online but can also outperform on-policy methods in more classic multi-goal RL environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。