用大模型显式实现高效探索算法，提升自然语言任务中的决策效率
Toward Efficient Exploration by Large Language Model Agents
- 让大模型直接实现已知的高效探索算法（后验采样强化学习）
- 在需要谨慎探索的语言任务中，表现优于现有大模型代理方法
- 无需微调或上下文学习，直接编程式调用经典强化学习机制
强化学习领域中，基于大语言模型（LLM）的序列决策代理正迅速发展。尽管这些自主决策代理有望推动诸多现实应用，但其成功依赖于数据高效的强化学习能力。探索效率是实现数据高效的关键障碍，而近期多数大模型代理设计对此缺乏有效应对。经典强化学习算法虽能优雅处理探索问题，但在纯自然语言环境中难以落地。本文不依赖微调或上下文学习来让大模型隐式模仿强化学习算法，而是直接显式实现一种已被广泛研究、具有统计高效探索能力的算法——后验采样强化学习（Posterior Sampling for Reinforcement Learning）。实验表明，该方法在需审慎探索的自然语言任务中，显著优于现有大模型代理方案。
原文摘要 · Abstract (English)
A burgeoning area within reinforcement learning (RL) is the design of sequential decision-making agents centered around large language models (LLMs). While autonomous decision-making agents powered by modern LLMs could facilitate numerous real-world applications, such successes demand agents that are capable of data-efficient RL. One key obstacle to achieving data efficiency in RL is exploration, a challenge that we demonstrate many recent proposals for LLM agent designs struggle to contend with. Meanwhile, classic algorithms from the RL literature known to gracefully address exploration require technical machinery that can be challenging to operationalize in purely natural language settings. In this work, rather than relying on finetuning or in-context learning to coax LLMs into implicitly imitating a RL algorithm, we illustrate how LLMs can be used to explicitly implement an existing RL algorithm (Posterior Sampling for Reinforcement Learning) whose capacity for statistically-efficient exploration is already well-studied. We offer empirical results demonstrating how our LLM-based implementation of a known, data-efficient RL algorithm can be considerably more effective in natural language tasks that demand prudent exploration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。