用大模型建议增强观察，让强化学习更高效探索复杂任务
Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
- 将大模型的动作建议作为增强观察，软性引导探索
- 在最复杂任务中成功率提升71%，训练速度最快快9倍
- 无需改算法，适合需高效探索的复杂强化学习场景
强化学习代理在稀疏奖励环境中常因传统探索策略失效而难以发现有效动作序列。大型语言模型（LLMs）具备从文本预训练中获得的程序性知识和推理能力，可辅助强化学习探索，但现有方法导致代理必须遵循大模型建议或直接将建议纳入奖励函数，缺乏灵活性。本文提出一种框架，通过增强观察空间提供大模型生成的动作建议，使强化学习代理能自主决定何时采纳或忽略该指导。该方法结合大模型的世界知识与推理能力，同时通过软约束保持灵活性。我们在三个难度递增的BabyAI环境中评估该方法，结果表明大模型指导的优势随任务难度增加而提升。在最复杂的环境中，最终成功率相比基线提高71%；样本效率显著提升，性能阈值达成速度最快快9倍，且无需修改现有强化学习算法。结果表明，该方法有效利用大模型规划能力，加速复杂环境下的强化学习训练。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) agents often struggle in sparse-reward environments where traditional exploration strategies fail to discover effective action sequences. Large Language Models (LLMs) possess procedural knowledge and reasoning capabilities from text pretraining that could guide RL exploration, but existing approaches create rigid dependencies where RL policies must follow LLM suggestions or incorporate them directly into reward functions. We propose a framework that provides LLM-generated action recommendations through augmented observation spaces, allowing RL agents to learn when to follow or ignore this guidance. Our method leverages LLMs' world knowledge and reasoning abilities while maintaining flexibility through soft constraints. We evaluate our approach on three BabyAI environments of increasing complexity and show that the benefits of LLM guidance scale with task difficulty. In the most challenging environment, we achieve 71% relative improvement in final success rates over baseline. The approach provides substantial sample efficiency gains, with agents reaching performance thresholds up to 9 times faster, and requires no modifications to existing RL algorithms. Our results demonstrate an effective method for leveraging LLM planning capabilities to accelerate RL training in challenging environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。