用提示词让大模型当强化学习智能体，零微调就能玩转简单游戏。
PARL: Prompt-based Agents for Reinforcement Learning
- 通过提示词编码状态、动作和奖励，让大模型在交互中学习
- 在简单环境中表现媲美甚至超过传统强化学习算法
- 适合非语言类推理任务，但复杂数学运算能力有限
大语言模型（LLMs）在自然语言表达的任务中表现出色，尤其在零样本或少样本设置下。这类任务通常被建模为监督（如分类）或无监督（如聚类）问题。然而，针对大模型作为强化学习（RL）智能体的应用研究较少，尤其是在需要与环境交互并基于奖励学习的游戏任务中。现有工作多聚焦于依赖语言表征的任务，而本文关注结构化、非语言的推理，如网格世界中的位置理解。为此，我们提出PARL（Prompt-based Agent for Reinforcement Learning），一种无需微调即可使用大模型作为强化学习智能体的方法。PARL通过提示词编码动作、状态和奖励，使模型能够通过试错进行学习。我们在三个不完全依赖自然语言的标准强化学习任务上评估了PARL，结果表明其在简单环境中可达到或超越传统强化学习智能体的表现，得益于预训练知识的利用。然而，在需要复杂数学运算或状态/动作解码的任务中，性能受限。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated high performance on tasks expressed in natural language, particularly in zero- or few-shot settings. These are typically framed as supervised (e.g., classification) or unsupervised (e.g., clustering) problems. However, limited work evaluates LLMs as agents in reinforcement learning (RL) tasks (e.g., playing games), where learning occurs through interaction with an environment and a reward system. While prior work focused on representing tasks that rely on a language representation, we study structured, non-linguistic reasoning - such as interpreting positions in a grid world. We therefore introduce PARL (Prompt-based Agent for Reinforcement Learning), a method that uses LLMs as RL agents through prompting, without any fine-tuning. PARL encodes actions, states, and rewards in the prompt, enabling the model to learn through trial-and-error interaction. We evaluate PARL on three standard RL tasks that do not entirely rely on natural language. We show that it can match or outperform traditional RL agents in simple environments by leveraging pretrained knowledge. However, we identify performance limitations in tasks that require complex mathematical operations or decoding states and actions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。