让智能体在线重放过往经验，提升学习效率。
True Online TD-Replan(lambda) Achieving Planning through Replaying
- 基于真在线TD算法，支持按顺序重放历史经验。
- 在随机游走和肌电控制任务中表现优于现有方法。
- 适合需要高效经验重用的强化学习场景。
本文提出一种新规划方法——真在线TD-重放(λ)，将真在线TD扩展为可在线按实际顺序重放全部或部分历史经验，频率由λ参数控制。该方法中λ不仅决定目标更新深度，还调节重放密度。实验表明，在受益于经验重放的问题上，该方法性能优于真在线TD(λ)，尽管复杂度为二次级；同时优于其他具有相似复杂度的方法（如Dyna规划、TD(λ)-重放）。测试环境包括使用二元特征的随机游走问题，以及结合表面肌电（sEMG）与深度提取特征的肌电控制任务，验证了其有效性。
原文摘要 · Abstract (English)
In this paper, we develop a new planning method that extends the capabilities of the true online TD to allow an agent to efficiently replay all or part of its past experience, online in the sequence that they appear with, either in each step or sparsely according to the usual λ parameter. In this new method that we call True Online TD-Replan(λ), the λ parameter plays a new role in specifying the density of the replay process in addition to the usual role of specifying the depth of the target's updates. We demonstrate that, for problems that benefit from experience replay, our new method outperforms true online TD(λ), albeit quadratic in complexity due to its replay capabilities. In addition, we demonstrate that our method outperforms other methods with similar quadratic complexity such as Dyna Planning and TD(λ)-Replan algorithms. We test our method on two benchmarking environments, a random walk problem that uses simple binary features and a myoelectric control domain that uses both simple sEMG features and deeply extracted features to showcase its capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。