用语言描述动作掩码,让大模型在强化学习中更可控地探索棋局。
VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training -- A Chess Case Study
- 通过提示词显式约束模型只从限定动作集中选择,实现可控制探索。
- 在国际象棋任务中,相比基线方法,平均每步失误降低15.3%,最终胜率提升21%。
- 适合研究大模型强化学习后训练的开发者,尤其关注探索效率问题。
强化学习(RL)后训练大语言模型(LLM)时,稀疏反馈和庞大的动作空间常导致模型过早陷入重复行为。本文提出语言化动作掩码(VAM),将动作掩码以自然语言形式融入提示,强制模型仅从掩码集合中输出动作。基于此接口,引入迭代动作空间剪枝:若目标动作未被采样,则从掩码中移除已采样的有效动作,缩小候选集后重新采样,直至采到目标动作或达到预设预算。我们在国际象棋场景下评估VAM,采用两种训练范式:一是与引擎对弈生成状态的引擎对弈制,二是基于固定位置数据集并使用验证器评分的固定数据集制。在保留棋局和完整对局中的平均中心分损失(ACPL)指标上,VAM显著优于多个强基线,证明语言化掩码是大模型强化学习后训练中实现可控探索的有效机制。
原文摘要 · Abstract (English)
Exploration remains a key bottleneck for reinforcement learning (RL) post-training of large language models (LLMs), where sparse feedback and large action spaces can lead to premature collapse into repetitive behaviors. We propose Verbalized Action Masking (VAM), which verbalizes an action mask in the prompt and enforces that the model outputs an action from the masked set. Building on this interface, we introduce iterative action-space pruning: if the target action is not sampled, we remove valid sampled actions from the mask and resample under the reduced candidate set, repeating until the target is sampled or a fixed budget is exhausted. We study VAM in chess and evaluate it under two training regimes: an engine-play regime that generates states via play against an engine opponent and a fixed-dataset regime that trains from a fixed dataset of positions with verifier scores. Across held-out chess puzzles and full-game play measured by average centipawn loss (ACPL), VAM improves learning efficiency and final performance over strong baselines, highlighting verbalized masking as a practical mechanism for controllable exploration in LLM RL post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。