提出新方法让强化学习同时满足收益与安全约束，效果优于传统方案。
Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations
- 基于哈密顿-雅可比方程构建双目标价值函数，显式求解约束问题。
- 在多任务场景中，成功率、安全性与速度均超越多个基线模型。
- 适合需要兼顾性能与安全的强化学习应用，如自动驾驶、机器人控制。
强化学习中的硬性约束常导致策略性能下降。拉格朗日方法虽能融合目标与约束，但需复杂的奖励设计与参数调优。本文将哈密顿-雅可比（HJ）方程与强化学习结合，提出两种新型价值函数以解决双目标满足问题：1）达始终避（RAA）问题——实现不同的奖励与惩罚阈值；2）达达（RR）问题——实现两个不同奖励的阈值。与依赖自动机表示的时序逻辑方法不同，本文通过分解推导出显式、可计算的贝尔曼形式。我们证明，RAA与RR问题可重写为先前研究的HJ-RL问题的组合。基于此分析，提出一种近端策略优化变体（DOHJ-PPO），在安全到达与多目标达成任务中，其表现优于多个基线，在成功率、安全性和速度上均有提升。
原文摘要 · Abstract (English)
Hard constraints in reinforcement learning (RL) often degrade policy performance. Lagrangian methods offer a way to blend objectives with constraints, but require intricate reward engineering and parameter tuning. In this work, we extend recent advances that connect Hamilton-Jacobi (HJ) equations with RL to propose two novel value functions for dual-objective satisfaction. Namely, we address: 1) the Reach-Always-Avoid (RAA) problem -- of achieving distinct reward and penalty thresholds -- and 2) the Reach-Reach (RR) problem -- of achieving thresholds of two distinct rewards. In contrast with temporal logic approaches, which typically involve representing an automaton, we derive explicit, tractable Bellman forms in this context via decomposition. Specifically, we prove that the RAA and RR problems may be rewritten as compositions of previously studied HJ-RL problems. We leverage our analysis to propose a variation of Proximal Policy Optimization (DOHJ-PPO), and demonstrate that it produces distinct behaviors from previous approaches, outcompeting a number of baselines in success, safety and speed across a range of tasks for safe-arrival and multi-target achievement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。