提出双最优策略评估方法，显著降低强化学习评估方差。
Doubly Optimal Policy Evaluation for Reinforcement Learning
- 设计数据收集与处理的最优组合策略
- 理论证明方差低于现有最佳方法
- 实测大幅降方差，适合高精度评估场景
策略评估通过（1）从环境收集数据和（2）将原始数据处理为有意义的估计来衡量策略性能。由于强化学习的序列特性，任何不当的数据收集策略或数据处理方法都会在长时步下显著增加评估结果的方差。因此，策略评估常面临大方差问题，需大量数据才能达到所需精度。本文设计了数据收集策略与数据处理基线的最优组合。理论上，我们证明所提出的双最优策略评估方法是无偏的，且方差严格低于此前表现最佳的方法。实证上，相比已有工作，我们的方法显著降低了方差并实现了更优的实验性能。
原文摘要 · Abstract (English)
Policy evaluation estimates the performance of a policy by (1) collecting data from the environment and (2) processing raw data into a meaningful estimate. Due to the sequential nature of reinforcement learning, any improper data-collecting policy or data-processing method substantially deteriorates the variance of evaluation results over long time steps. Thus, policy evaluation often suffers from large variance and requires massive data to achieve the desired accuracy. In this work, we design an optimal combination of data-collecting policy and data-processing baseline. Theoretically, we prove our doubly optimal policy evaluation method is unbiased and guaranteed to have lower variance than previously best-performing methods. Empirically, compared with previous works, we show our method reduces variance substantially and achieves superior empirical performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。