提出新型估计器,显著降低离策略评估与学习中的方差和敏感性。
Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning
- 基于对数求和指数运算构造新估计器,提升稳定性。
- 在重尾奖励下实现方差压缩,理论证明收敛率达O(n^{-ε/(1+ε)})。
- 适用于高噪声、低质量倾向得分场景,适合强化学习应用者。
离策略学习与评估利用记录的上下文-动作-倾向得分-反馈数据集,但常面临高方差、倾向得分质量差及重尾奖励分布等问题。本文提出一种基于对数求和指数(LSE)算子的新估计器,显著优于传统逆倾向得分方法。该方法在重尾条件下表现出更强的鲁棒性和方差抑制能力。对于离策略评估,我们推导了估计器偏差与方差的上界;在离策略学习中,假设加权奖励具有有界(1+ε)阶矩,建立了关于后悔(性能差距)的理论边界,其收敛速率为O(n^{-ε/(1+ε)}),其中ε∈[0,1],n为数据集规模。理论分析得到全面实证验证,涵盖评估与学习双场景,证实方法的实用性。代码已公开于https://github.com/armin-behnamnia/lse-offpolicy-learning。
原文摘要 · Abstract (English)
Off-policy learning and evaluation leverage logged bandit feedback datasets, which contain context, action, propensity score, and feedback for each data point. These scenarios face significant challenges due to high variance and poor performance with low-quality propensity scores and heavy-tailed reward distributions. We address these issues by introducing a novel estimator based on the log-sum-exponential (LSE) operator, which outperforms traditional inverse propensity score estimators. Our LSE estimator demonstrates variance reduction and robustness under heavy-tailed conditions. For off-policy evaluation, we derive upper bounds on the estimator's bias and variance. In the off-policy learning scenario, we establish bounds on the regret -- the performance gap between our LSE estimator and the optimal policy -- assuming bounded $(1+ε)$-th moment of weighted reward. Notably, we achieve a convergence rate of $O(n^{-ε/(1+ ε)})$ for the regret bounds, where $ε\in [0,1]$ and $n$ is the size of logged bandit feedback dataset. Theoretical analysis is complemented by comprehensive empirical evaluations in both off-policy learning and evaluation scenarios, confirming the practical advantages of our approach. The code for our estimator is available at the following link: https://github.com/armin-behnamnia/lse-offpolicy-learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。