深度强化学习中，两种时间差分误差定义不再等价，影响算法性能。
Deep Reinforcement Learning and The Tale of Two Temporal Difference Errors
- 区分两种TD误差：前后预测差与自举目标差
- 深度网络非线性越强，两种误差差异越大
- 为平均奖励RL等方法提供新优化思路
时间差分（TD）误差最初由Sutton（1988）提出，被描述为连续预测之间的差异，随后又定义为自举目标与预测之间的差异。此后，这两种解释在文献中被交替使用，后者成为深度强化学习（RL）架构中的标准评论器损失。本文表明，这两种对TD误差的解释在深层非线性模型中并不总是等价。具体而言，随着深度强化学习架构的非线性增强，两种解释产生的数值差异逐渐增大。基于此发现，我们进一步揭示了选择其中一种解释会影响依赖于TD误差计算其他量的深度强化学习算法的表现，如深度微分（即平均奖励）强化学习方法。总体而言，我们的结果表明，在深度强化学习设置下，将TD误差默认视为自举目标与预测之差的做法并不总成立。
原文摘要 · Abstract (English)
The temporal difference (TD) error was first formalized in Sutton (1988), where it was first characterized as the difference between temporally successive predictions, and later, in that same work, formulated as the difference between a bootstrapped target and a prediction. Since then, these two interpretations of the TD error have been used interchangeably in the literature, with the latter eventually being adopted as the standard critic loss in deep reinforcement learning (RL) architectures. In this work, we show that these two interpretations of the TD error are not always equivalent. In particular, we show that increasingly-nonlinear deep RL architectures can cause these interpretations of the TD error to yield increasingly different numerical values. Then, building on this insight, we show how choosing one interpretation of the TD error over the other can affect the performance of deep RL algorithms that utilize the TD error to compute other quantities, such as with deep differential (i.e., average-reward) RL methods. All in all, our results show that the default interpretation of the TD error as the difference between a bootstrapped target and a prediction does not always hold in deep RL settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。