改进TD(λ)算法，让智能体更关注状态间相对优劣。
Reinforcement Learning From State and Temporal Differences
- 用相对状态值替代绝对值进行函数逼近
- 在双状态和三状态系统中实现策略单调提升
- 适用于需精确排序状态的决策任务
TD(λ)结合函数逼近在复杂强化学习问题中表现良好。对于线性逼近，它最小化各状态近似值与真实值之间的平方误差。然而，策略优化的关键在于状态间的相对排序误差，而非绝对值误差。本文通过双状态、三状态系统及西洋双陆棋案例证明，即使从最优策略出发，标准TD(λ)仍可能收敛至次优策略。为此提出新算法STD(λ)，其函数逼近器基于二元决策问题中的相对状态值进行训练。理论分析给出了两状态系统下策略单调改善的证明，并与Bertsekas的差分训练方法比较。实验证明，STD(λ)在双状态系统和改良版acrobot问题上均取得成功。
原文摘要 · Abstract (English)
TD($λ$) with function approximation has proved empirically successful for some complex reinforcement learning problems. For linear approximation, TD($λ$) has been shown to minimise the squared error between the approximate value of each state and the true value. However, as far as policy is concerned, it is error in the relative ordering of states that is critical, rather than error in the state values. We illustrate this point, both in simple two-state and three-state systems in which TD($λ$)--starting from an optimal policy--converges to a sub-optimal policy, and also in backgammon. We then present a modified form of TD($λ$), called STD($λ$), in which function approximators are trained with respect to relative state values on binary decision problems. A theoretical analysis, including a proof of monotonic policy improvement for STD($λ$) in the context of the two-state system, is presented, along with a comparison with Bertsekas' differential training method [1]. This is followed by successful demonstrations of STD($λ$) on the two-state system and a variation on the well known acrobot problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。