用游戏解法器的决策价值变化，给大模型提供每一步的奖励信号。
CAST: Game Solvers as Turn-Level Teachers for LLM Agents
- 通过解法器状态价值变化生成每步奖励信号
- 在三类游戏中均超越基线，零样本表现最优
- 适合需要长序列决策的强化学习任务
让大语言模型在长时序游戏中决策是迈向通用决策的重要一步，但基于可验证奖励的强化学习（RLVR）依赖稀疏的最终奖励，难以判断哪些决策决定成败。更密集的过程信号能弥补这一缺陷，但现有方法要么成本高，要么不准确。我们发现游戏解法器的状态价值变化可反映动作是否推动状态向成功靠近。基于此提出CAST（来自解法器教师的信用分配），将价值变化转化为解法器优势，并作为每步信号注入到RLVR中。进一步证明，在软最优解法器假设下，最大化解法器优势等价于从解法器进行策略蒸馏，仅需标量值而非教师输出分布。在Sokoban、Minesweeper和Rush Hour上，CAST在所有游戏的域内及未见难度评估中均优于所有训练基线，并在ALFWorld和WebShop上取得最高平均零样本性能。代码已开源。
原文摘要 · Abstract (English)
Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。