arXiv:2411.00405stat.MLcs.LG2024-11

提出HAVER算法,精准估计多分布最大均值,优于传统方法。

HAVER: Instance-Dependent Error Bounds for Maximum Mean Estimation and Applications to Q-Learning and Monte Carlo Tree Search

  • 通过头平均策略融合样本信息,优化最大均值估计
  • 误差率优于已知最优的基准算法,尤其在近优分布多时表现更佳
  • 适用于强化学习中的Q-learning与蒙特卡洛树搜索

我们研究从K个分布中采样以估计其最大均值的问题(而非判断哪个分布最大),该问题出现在Q-learning和蒙特卡洛树搜索(MCTS)等机器学习任务中。现有算法的性能分析仅限于偏差,缺乏对精确误差度量的探讨。本文提出新算法HAVER(Head AVERaging),并分析其均方误差。理论表明:1)HAVER的性能媲美已知最优分布身份的“神谕”(oracle),后者直接报告最佳分布的样本均值;2)令人意外的是,当存在大量接近最优的分布时,HAVER的收敛速率甚至优于该神谕。这两项改进均为文献首例,且证明简单取最大经验均值的朴素算法无法达到此界。数值实验验证了理论结果:在多臂赌博机、Q-learning和MCTS中,HAVER持续优于基线方法,展现跨场景有效性。

原文摘要 · Abstract (English)

We study the problem of estimating the \emph{value} of the largest mean among K distributions via samples from them (rather than estimating \emph{which} distribution has the largest mean), which arises from various machine learning tasks including Q-learning and Monte Carlo Tree Search (MCTS). While there have been a few proposed algorithms, their performance analyses have been limited to their biases rather than a precise error metric. In this paper, we propose a novel algorithm called HAVER (Head AVERaging) and analyze its mean squared error. Our analysis reveals that HAVER has a compelling performance in two respects. First, HAVER estimates the maximum mean as well as the oracle who knows the identity of the best distribution and reports its sample mean. Second, perhaps surprisingly, HAVER exhibits even better rates than this oracle when there are many distributions near the best one. Both of these improvements are the first of their kind in the literature, and we also prove that the naive algorithm that reports the largest empirical mean does not achieve these bounds. Finally, we confirm our theoretical findings via numerical experiments where we implement HAVER in bandit, Q-learning, and MCTS algorithms. In these experiments, HAVER consistently outperforms the baseline methods, demonstrating its effectiveness across different applications.

强化学习统计估计最优决策

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。