提出Q-MMR方法，实现无需复杂假设的离线强化学习评估。
Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching
- 通过递归加权与矩匹配学习数据权重，逼近目标策略回报。
- 在仅需Q^π可表示条件下，实现无维度误差的有限样本保证。
- 揭示覆盖性本质，适合研究离线强化学习理论的学者。
我们提出一种新的理论框架Q-MMR，用于有限时域马尔可夫决策过程中的离线策略评估。Q-MMR学习一组标量权重（每个数据点一个），使重加权后的奖励逼近目标策略下的期望回报。权重通过自顶向下的递归方式，基于值函数判别器类的矩匹配目标进行学习。值得注意的是，即使在一般函数逼近下，仅需Q^π可实现性这一条件，即可建立数据依赖的有限样本保证，且误差不依赖函数类的统计复杂度（维度无关）。我们还建立了该方法与重要性采样、线性FQE等现有方法的联系。进一步的理论分析揭示了覆盖性这一离线强化学习核心概念的本质。
原文摘要 · Abstract (English)
We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-down manner via a moment matching objective against a value-function discriminator class. Notably, and perhaps surprisingly, a data-dependent finite-sample guarantee for general function approximation can be established under only the realizability of $Q^π$, with a dimension-free bound -- that is, the error does not depend on the statistical complexity of the function class. We also establish connections to several existing methods, such as importance sampling and linear FQE. Further theoretical analyses shed new light on the nature of coverage, a concept of fundamental importance to offline RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。