让大模型规划动作，用强化学习不断修正错误，提升智能体执行能力。
SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding
- 大模型生成动作计划，强化学习训练具体执行策略并反馈结果迭代优化。
- 在Craftax任务中收集钻石成功率88.2%，比最佳基线提升1.9倍。
- 适合需要高可靠性任务规划的智能体研究者，尤其关注错误修正机制。
基于语言模型的智能体在拥有高层动作API时表现优异，但难以将语言指令转化为底层控制。以往工作让大模型一次性生成技能或奖励函数，缺乏反馈来纠正规范错误。本文提出SCALAR，一种双向框架，通过学习到的技能库将大模型规划与强化学习结合。大模型生成带前提和效果的技能；强化学习训练每个技能的策略，并将执行结果反馈以迭代优化规格，增强对初始错误的鲁棒性。关键轨迹分析通过分析强化学习轨迹修正大模型先验；前沿检查点可选地在技能边界保存环境状态，提升样本效率。在Craftax上，SCALAR实现88.2%的钻石收集率，是最佳基线的1.9倍，且能9.1%概率抵达精灵矿洞，而之前方法完全失败。
原文摘要 · Abstract (English)
LM-based agents excel when given high-level action APIs but struggle to ground language into low-level control. Prior work has LLMs generate skills or reward functions for RL, but these one-shot approaches lack feedback to correct specification errors. We introduce SCALAR, a bidirectional framework coupling LLM planning with RL through a learned skill library. The LLM proposes skills with preconditions and effects; RL trains policies for each skill and feeds back execution results to iteratively refine specifications, improving robustness to initial errors. Pivotal Trajectory Analysis corrects LLM priors by analyzing RL trajectories; Frontier Checkpointing optionally saves environment states at skill boundaries to improve sample efficiency. On Craftax, SCALAR achieves 88.2% diamond collection, a 1.9x improvement over the best baseline, and reaches the Gnomish Mines 9.1% of the time where prior methods fail entirely.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。