用大模型指导强化学习，让智能体更快学会复杂动作。
Efficient Soft Actor-Critic with LLM-Based Action-Level Guidance for Continuous Control
- 用大模型分析状态和视觉回放，实时给出动作建议
- 在多个环境中样本效率提升，最终得分超越SAC等方法
- 适合需要高效探索的连续控制任务，如机器人操控
我们提出GuidedSAC，一种新型强化学习算法，用于在巨大状态-动作空间中实现高效探索。GuidedSAC利用大语言模型（LLM）作为智能监督者，为软演员-评论家（SAC）算法提供动作级指导。该监督者通过分析最近轨迹的状态信息与视觉回放，生成动作级干预，实现有目标的探索。此外，我们对GuidedSAC进行了理论分析，证明其在保持SAC收敛性的同时提升了收敛速度。在离散与连续控制环境（包括玩具文本任务和复杂的MuJoCo基准）中的实验表明，GuidedSAC在样本效率和最终性能上均持续优于标准SAC及最先进的探索增强变体（如RND、ICM和E3B）。
原文摘要 · Abstract (English)
We present GuidedSAC, a novel reinforcement learning (RL) algorithm that facilitates efficient exploration in vast state-action spaces. GuidedSAC leverages large language models (LLMs) as intelligent supervisors that provide action-level guidance for the Soft Actor-Critic (SAC) algorithm. The LLM-based supervisor analyzes the most recent trajectory using state information and visual replays, offering action-level interventions that enable targeted exploration. Furthermore, we provide a theoretical analysis of GuidedSAC, proving that it preserves the convergence guarantees of SAC while improving convergence speed. Through experiments in both discrete and continuous control environments, including toy text tasks and complex MuJoCo benchmarks, we demonstrate that GuidedSAC consistently outperforms standard SAC and state-of-the-art exploration-enhanced variants (e.g., RND, ICM, and E3B) in terms of sample efficiency and final performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。