用大模型动态生成策略，让强化学习更高效可解释
DYSTIL: Dynamic Strategy Induction with Large Language Models for Reinforcement Learning
- 让大模型根据优势估计和专家示范生成文本策略
- 在Minigrid和BabyAI上成功率比基线高17.75%，样本效率更高
- 策略演变过程可读可查，适合需要可解释性的场景
从专家示范中进行强化学习长期是难题，现有基于行为克隆加后续强化学习训练的方法常存在泛化能力差、样本效率低和模型不可解释等问题。受大语言模型强大推理能力启发，我们提出一种结合大模型的新型策略型强化学习框架 DYSTIL（DYnamic STrategy Induction with Llms for reinforcement learning），以克服这些局限。DYSTIL 动态调用策略生成型大模型，基于优势估计与专家示范生成文本策略，并通过策略优化逐步将诱导出的策略内化至强化学习智能体中，从而提升性能，增强策略泛化性并提高样本效率。同时，该方法提供直接的文本通道，可观测训练过程中策略演化的全过程。我们在 Minigrid 与 BabyAI 的复杂强化学习环境中测试了 DYSTIL，实证表明其平均成功率相比最优基线方法显著提升 17.75%，且学习过程具有更高的样本效率。
原文摘要 · Abstract (English)
Reinforcement learning from expert demonstrations has long remained a challenging research problem, and existing state-of-the-art methods using behavioral cloning plus further RL training often suffer from poor generalization, low sample efficiency, and poor model interpretability. Inspired by the strong reasoning abilities of large language models (LLMs), we propose a novel strategy-based reinforcement learning framework integrated with LLMs called DYnamic STrategy Induction with Llms for reinforcement learning (DYSTIL) to overcome these limitations. DYSTIL dynamically queries a strategy-generating LLM to induce textual strategies based on advantage estimations and expert demonstrations, and gradually internalizes induced strategies into the RL agent through policy optimization to improve its performance through boosting policy generalization and enhancing sample efficiency. It also provides a direct textual channel to observe and interpret the evolution of the policy's underlying strategies during training. We test DYSTIL over challenging RL environments from Minigrid and BabyAI, and empirically demonstrate that DYSTIL significantly outperforms state-of-the-art baseline methods by 17.75% in average success rate while also enjoying higher sample efficiency during the learning process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。