提出随机决策时域框架,实现每步约束满足的强化学习。
Stochastic Decision Horizons for Constrained Reinforcement Learning
- 用状态-动作继续概率动态缩短时域,约束违规即缩减决策范围。
- 在Hyfydy上训练效率提升4倍,且比现有方法更稳定。
- 适合对安全要求高、需实时满足约束的机器人控制场景。
我们提出随机决策时域(SDH),一种理论完备的框架,用于解决每步约束满足的受限强化学习问题,这在诸多现实应用中至关重要。在SDH中,约束违规会通过状态-动作继续概率导致有效时域缩短。基于Control as Inference,我们开发了首个支持离策略与正则化的即时约束强化学习算法。我们明确了两种违规后决策的合理语义:吸收态语义终止决策过程,仅存活决策承担熵成本,得到最大熵吸收态SAC(AS-SAC);虚拟终止保持决策延续但停止奖励信用,得到KL正则化虚拟终止MPO(VT-MPO)。为连接SDH与约束马尔可夫决策过程(CMDP),我们追踪轨迹上违规的累积深度(违规深度分布)。SDH对每条轨迹按违规总量的指数加权,当违规集中在单一尺度时,等价于添加型CMDP预算;我们指出了其失效情形:当罕见深层违规与频繁浅层违规共存时。实验验证了理论。在90肌肉的H2190人形机器人(Hyfydy)上,VT-MPO以4倍更少环境步数达到顶尖步态真实度,并显著提升训练稳定性。在Safety Gymnasium上,违规深度分布准确识别出SDH能提供强收益-违规权衡的场景。
原文摘要 · Abstract (English)
We propose stochastic decision horizons (SDH), a theoretically grounded framework for solving constrained RL problems with every-step constraint satisfaction, a desirable property in many real-world applications. In SDH, a constraint violation yields an effective shortening of horizon via a state-action continuation probability. Using Control as Inference, we develop the first off-policy and regularized algorithms for RL with instantaneous constraints. We identify two principled semantics for what counts as a decision after a violation. Absorbing-state semantics end the decision process, so only surviving decisions pay entropy cost, yielding max-entropy AS-SAC. Virtual-termination keeps the decision process alive while stopping reward credit, yielding KL-regularized VT-MPO. To connect SDH with CMDPs, we track how violations accumulate along trajectories (their violation-depth profile). SDH effectively weights each trajectory by the exponential of its total violation; this matches an additive CMDP budget exactly when violations occur at a single characteristic scale, and we pinpoint where it cannot: when rare, deep violations mix with frequent, shallow ones. Experiments validate the theory. On the 90-muscle H2190 humanoid (Hyfydy), VT-MPO matches state-of-the-art gait realism with $4\times$ fewer environment steps and substantially more stable training. On Safety Gymnasium, violation-depth profiles correctly identify the regimes in which SDH delivers strong reward-violation trade-offs. Experiments validate the theory. On the 90-muscle H2190 humanoid (Hyfydy), VT-MPO matches state-of-the-art gait realism with 4x fewer environment steps and substantially more stable training. On Safety Gymnasium, violation-depth profiles correctly identify the regimes in which SDH delivers strong reward-violation trade-offs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。