让强化学习更新更稳定,每步都按预期效果自动调步长。
Intentional Updates for Streaming Reinforcement Learning
- 根据期望的更新效果反推步长,而非固定参数步长。
- 在单步流式训练中实现与批量方法相当的性能。
- 适合对稳定性要求高的在线强化学习场景。
在基于梯度的学习中,以参数单位设定的学习率无法预测每步对函数输出的影响,常导致流式设置(即批大小=1)下的不稳定,因随机性未被平均,更新幅度可能瞬间过大或过小。为此,我们提出有意更新:先指定更新期望结果,再求解近似达成该结果的步长。这一思想源自在线监督线性回归中的归一化最小均方算法,其通过步长选择实现与当前误差成比例的输出变化。我们将其扩展至流式深度强化学习,定义了恰当的预期目标:有意TD旨在实现固定的TD误差降低比例,有意策略梯度旨在限制每步策略变化,从而控制局部KL散度。我们提出了结合资格迹和对角缩放的实际算法。实验表明,这些方法在流式环境下达到领先性能,频繁表现与批量及回放缓冲区方法相当。
原文摘要 · Abstract (English)
In gradient-based learning, a step size chosen in parameter units does not produce a predictable per-step change in function output. This often leads to instability in the streaming setting (i.e., batch size=1), where stochasticity is not averaged out and update magnitudes can momentarily become arbitrarily big or small. Instead, we propose intentional updates: first specify the intended outcome of an update and then solve for the step size that approximately achieves it. This strategy has precedent in online supervised linear regression via Normalized Least Mean Squares algorithm, which selects a step size to yield a specified change in the function output proportional to the current error. We extend this principle to streaming deep reinforcement learning by defining appropriate intended outcomes: Intentional TD aims for a fixed fractional reduction of the TD error, and Intentional Policy Gradient aims for a bounded per-step change in the policy, limiting local KL divergence. We propose practical algorithms combining eligibility traces and diagonal scaling. Empirically, these methods yield state-of-the-art streaming performance, frequently performing on par with batch and replay-buffer approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。