提出新算法与实验协议，提升离线强化学习中策略评估的准确性。
Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
- 设计有理论保证的模型自由与模型依赖选择器。
- 新协议实现更稳定的候选值函数生成与无优化控制。
- 适合研究离线RL评估方法的学者与工程师参考。
在离线强化学习中，从数据中进行留出验证和超参数调优是长期存在的难题。标准框架使用离策略评估（OPE）方法来评估和选择策略，但现有OPE方法要么方差呈指数增长（如重要性采样），要么自身带有超参数（如FQE和基于模型的方法）。本文聚焦于OPE自身的超参数调优问题，这是更少被研究的方向。具体而言，我们在候选值函数（‘模型自由’）或动态模型（‘模型依赖’）中进行选择，以最佳评估目标策略的性能。我们提出了：(1) 具有理论保证的新模型自由与模型依赖选择器；(2) 一种新的实验协议，用于对这些选择器进行经验评估。相比以往工作中的模型自由协议，我们的新协议无需优化即可实现更稳定的候选值函数生成，并对候选值函数有更好的控制能力，同时支持对模型自由与模型依赖方法的统一评估。我们在Gym-Hopper上验证该协议，发现新提出的模型自由选择器LSTD-Tournament表现出色的实证性能。
原文摘要 · Abstract (English)
Holdout validation and hyperparameter tuning from data is a long-standing problem in offline reinforcement learning (RL). A standard framework is to use off-policy evaluation (OPE) methods to evaluate and select the policies, but OPE either incurs exponential variance (e.g., importance sampling) or has hyperparameters on their own (e.g., FQE and model-based). We focus on hyperparameter tuning for OPE itself, which is even more under-investigated. Concretely, we select among candidate value functions ("model-free") or dynamics ("model-based") to best assess the performance of a target policy. Concretely, we select among candidate value functions (``model-free'') or dynamics models (``model-based'') to best assess the performance of a target policy. We develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) a new experimental protocol for empirically evaluating them. Compared to the model-free protocol in prior works, our new protocol allows for more stable generation and better control of candidate value functions in an optimization-free manner, and evaluation of model-free and model-based methods alike. We exemplify the protocol on Gym-Hopper, and find that our new model-free selector, LSTD-Tournament, demonstrates promising empirical performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。