arXiv:2603.09090cs.LG2026-03被引 1

发现未掩码强化学习会抑制未访问状态的合法动作,提出解决方案。

Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms

  • 通过共享参数传播梯度,导致未访问状态合法动作概率被压制。
  • 合法动作概率受指数级衰减约束,熵正则化难以兼顾保护与效率。
  • 实验验证抑制现象并证明可行性分类可替代真实动作掩码。

在状态依赖动作有效性的强化学习环境中,动作掩码始终优于基于惩罚的方法,但现有理论仅证明掩码能保持策略梯度定理。我们识别出未掩码训练的一个独特失败模式:系统性抑制代理尚未访问状态中的合法动作。这是由于在已访问状态中无效动作的梯度通过共享网络参数传播至未访问状态,而这些状态中该动作实际合法。我们证明,对于具有共享特征的softmax策略,当某动作在已访问状态无效但在未访问状态$ s^* $有效时,其在$ s^* $处的概率$ π(a mid s^*) $受指数衰减约束,该约束源于参数共享和softmax对数几率的零和性质。这一边界揭示熵正则化在保护合法动作与样本效率之间存在权衡,而掩码方法可消除此权衡。我们实证验证了深度网络满足特征对齐条件,且在Craftax、Craftax-Classic和MiniHack上的实验确认了预测的指数抑制现象,并表明可行性分类可实现无需真掩码的部署。

原文摘要 · Abstract (English)

In reinforcement learning environments with state-dependent action validity, action masking consistently outperforms penalty-based handling of invalid actions, yet existing theory only shows that masking preserves the policy gradient theorem. We identify a distinct failure mode of unmasked training: it systematically suppresses valid actions at states the agent has not yet visited. This occurs because gradients pushing down invalid actions at visited states propagate through shared network parameters to unvisited states where those actions are valid. We prove that for softmax policies with shared features, when an action is invalid at visited states but valid at an unvisited state $s^*$, the probability $π(a \mid s^*)$ is bounded by exponential decay due to parameter sharing and the zero-sum identity of softmax logits. This bound reveals that entropy regularization trades off between protecting valid actions and sample efficiency, a tradeoff that masking eliminates. We validate empirically that deep networks exhibit the feature alignment condition required for suppression, and experiments on Craftax, Craftax-Classic, and MiniHack confirm the predicted exponential suppression and demonstrate that feasibility classification enables deployment without oracle masks.

强化学习动作掩码策略梯度有效性约束

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。