让AI自动识别无效动作，提升强化学习效率与安全性
Neuro-symbolic Action Masking for Deep Reinforcement Learning
- 用神经符号方法自动构建状态符号模型
- 显著提升采样效率并减少约束违反
- 适合需要安全决策的强化学习场景
深度强化学习（DRL）在训练和执行过程中可能探索不可行的动作。现有方法依赖人工指定的符号映射函数和动作掩码来约束行为。本文提出神经符号动作掩码（NSAM），一种在最小监督下自动学习与高维状态域约束一致的符号模型的新框架。基于学习到的状态符号模型，NSAM自动生成排除不可行动作的掩码。该方法实现了符号推理与深度策略优化的端到端融合，符号对齐与策略学习相互促进。我们在多个带约束的领域评估了NSAM，实验结果表明其显著提升了DRL代理的样本效率，并大幅减少约束违反。
原文摘要 · Abstract (English)
Deep reinforcement learning (DRL) may explore infeasible actions during training and execution. Existing approaches assume a symbol grounding function that maps high-dimensional states to consistent symbolic representations and a manually specified action masking techniques to constrain actions. In this paper, we propose Neuro-symbolic Action Masking (NSAM), a novel framework that automatically learn symbolic models, which are consistent with given domain constraints of high-dimensional states, in a minimally supervised manner during the DRL process. Based on the learned symbolic model of states, NSAM learns action masks that rules out infeasible actions. NSAM enables end-to-end integration of symbolic reasoning and deep policy optimization, where improvements in symbolic grounding and policy learning mutually reinforce each other. We evaluate NSAM on multiple domains with constraints, and experimental results demonstrate that NSAM significantly improves sample efficiency of DRL agent while substantially reducing constraint violations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。