针对多次尝试最优结果的奖励机制,提出新型强化学习算法并证明其可实现次线性后悔率。
Finite-Time Regret Analysis of Retry-Aware Bandits

- 基于后验期望最大值设计采样策略,平衡探索与利用
- 首次证明在高斯奖励下M=2时后悔率次线性,优于传统方法
- 揭示最优臂低估效应,适合对探索效率敏感的应用场景
我们研究了一种受重试意识目标启发的随机多臂赌博机算法,该目标重视多次尝试中的最佳结果,如pass@$k$和max@$k$。ReMax在给定臂值后验分布的情况下,选择一种采样分布,以最大化在M次虚拟抽样下的后验期望最大奖励。尽管这一目标最初在强化学习中被用作不确定性下的探索机制,但其在多臂赌博机问题中的后悔性质仍不明确。针对高斯奖励及首个非平凡情形M=2,我们通过期望改进平衡条件刻画了最优ReMax分布,并证明了ReMax的首个次线性后悔界。我们的分析将次优臂的通常饱和行为与ReMax特有的低估效应区分开来:当获得不利估计后,最优臂可能被采样过少。这解释了为何ReMax比汤普森采样更具开发性,且其后悔分析技术上更为复杂。实验支持这一观点:在轻微低估情况下,ReMax常优于KL-UCB和汤普森采样;而后验方差缩放在实践中可缓解严重低估问题。
原文摘要 · Abstract (English)
We study a stochastic bandit algorithm motivated by retry-aware objectives that value the best outcome among multiple attempts, such as pass@$k$ and max@$k$. Given a posterior over arm values, ReMax chooses a sampling distribution that maximizes the posterior expected maximum reward over $M$ virtual draws. Although this objective was introduced in reinforcement learning as an exploration mechanism under uncertainty, its regret properties in bandit problems have remained unclear. For Gaussian rewards and the first nontrivial case $M=2$, we characterize the optimal ReMax distribution through an expected-improvement balance condition and prove the first sublinear regret bound for ReMax. Our analysis separates the usual saturation behavior of suboptimal arms from a ReMax-specific underestimation effect, in which the optimal arm may be sampled too rarely after an unfavorable estimate. This explains why ReMax can be more exploitative than Thompson sampling (TS) and why its regret analysis is technically delicate. Experiments support this picture: ReMax often outperforms KL-UCB and Thompson sampling under mild underestimation, while posterior-variance scaling empirically mitigates severe underestimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。