改进受限动作空间的强化学习，用精确计算提升稳定性和效率。
Improving Stochastic Action-Constrained Reinforcement Learning via Truncated Distributions
- 用截断分布建模动作约束，避免近似带来的偏差
- 提出高效数值方法计算熵、对数概率及梯度，性能提升显著
- 适合需要安全或精准动作控制的RL应用
在强化学习中，对动作空间施加约束有助于保障安全或提高动作相关性。现有动作约束强化学习面临策略更新有效、计算效率和运行时可预测性等挑战。近期工作采用截断正态分布进行随机策略梯度方法，但在复杂约束下，熵、对数概率及其梯度的计算变得不可解析。此前工作通过非截断分布近似这些量，严重损害性能。本文认为准确估计这些特性对动作约束强化学习至关重要，提出了高效的数值近似方法，并设计了截断策略分布的高效采样策略。我们在三个基准环境上验证了该方法，结果表明使用精确估计可带来显著性能提升。
原文摘要 · Abstract (English)
In reinforcement learning (RL), it is often advantageous to consider additional constraints on the action space to ensure safety or action relevance. Existing work on such action-constrained RL faces challenges regarding effective policy updates, computational efficiency, and predictable runtime. Recent work proposes to use truncated normal distributions for stochastic policy gradient methods. However, the computation of key characteristics, such as the entropy, log-probability, and their gradients, becomes intractable under complex constraints. Hence, prior work approximates these using the non-truncated distributions, which severely degrades performance. We argue that accurate estimation of these characteristics is crucial in the action-constrained RL setting, and propose efficient numerical approximations for them. We also provide an efficient sampling strategy for truncated policy distributions and validate our approach on three benchmark environments, which demonstrate significant performance improvements when using accurate estimations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。