提出新算法,可精准生成多目标强化学习的最优权衡策略
Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning
- 基于切比雪夫加权设计偏好条件贝尔曼算子
- 收敛后覆盖整个帕累托前沿,且保证策略近似最优
- 适合需要精确权衡多个目标的决策场景
现实决策常需平衡多个冲突目标,传统强化学习通常将奖励合并为单一标量信号,难以捕捉完整的最优权衡关系,即帕累托前沿。本文提出一种基于切比雪夫加权的偏好条件贝尔曼算子,用于求解多目标马尔可夫决策过程(MOMDP)中的确定性帕累托最优策略。证明该算子具有包络性质,估计值函数上界真实帕累托前沿,并单调收敛至前沿的覆盖集。进一步展示如何从收敛的Q值估计中提取确定性策略,使智能体能针对任意偏好恢复对应策略,完整覆盖帕累托前沿,且每条合成策略均近似帕累托最优。实验验证算法能有效恢复复杂权衡关系,为确定性帕累托最优策略合成提供可行方案。
原文摘要 · Abstract (English)
Real-world decision-making often requires balancing multiple conflicting objectives, a challenge that standard Reinforcement Learning (RL) frequently addresses by aggregating rewards into a single scalar signal. While effective for simple tasks, this approach often fails to capture the full spectrum of optimal trade-offs, known as the Pareto frontier. In this paper, we introduce a novel preference-conditioned Bellman operator, motivated from the Chebyshev scalarization, designed to compute deterministic Pareto-optimal policies for Multi-Objective Markov Decision Processes (MOMDPs). We prove that this operator satisfies an enveloping property, where the estimated value functions upper-bound the true Pareto frontier, and demonstrate that it monotonically converges to a coverage set of this frontier. Furthermore, we also show how to extract deterministic policies from these converged Q-estimates. This ensures the agent can recover a policy for any given preference, capturing the entire Pareto-optimal frontier while guaranteeing each synthesized policy remains approximately Pareto-optimal. Experimental results validate that our algorithm successfully recovers complex trade-offs, providing a solution for deterministic Pareto-optimal policy synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。