用相对价值替代绝对价值，提升强化学习的控制效率
Relative Value Learning
- 直接学习状态间价值差，避免估计绝对值的冗余
- 在49个Atari游戏上性能媲美标准PPO，证明方法有效
- 适合追求高效策略梯度估计的强化学习研究者
在强化学习中，评判器通常估计绝对状态值 $V(s)$，衡量特定状态本身的好坏。但实际控制仅依赖于价值差异。为此，我们提出相对价值学习（RV），通过反对称函数 $Δ(s_i, s_j) = V(s_i) - V(s_j)$ 直接学习价值差。引入成对贝尔曼算子，证明其为 $γ$-收缩，具有唯一不动点等于真实价值差；推导出1步、n步及λ返回目标，并从成对差值重构广义优势估计（R-GAE），获得无偏策略梯度估计。理论之外，将RV与PPO结合，在49个Atari游戏上表现媲美标准PPO，表明相对价值估计是绝对评判器的有效替代。
原文摘要 · Abstract (English)
In reinforcement learning, critics typically estimate absolute state values $V(s)$, estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function $Δ(s_i, s_j) = V(s_i) - V(s_j)$. We introduce a pairwise Bellman operator and prove it is a $γ$-contraction with a unique fixed point equal to the true value differences, derive well-posed $1$-step, $n$-step and $λ$-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。