改进在线潘多拉盒子问题的后悔率,实现更优学习性能。
Improved Regret and Contextual Linear Extension for Pandora's Box and Prophet Inequality
- 设计新算法,实现$ ilde{O}( oot{nT}$)后悔率
- 在上下文线性场景下达到$ ilde{O}(nd oot{T})$后悔率
- 适用于在线决策与资源选择场景
我们研究带有半-赌博反馈的在线学习设定下的潘多拉盒子问题。每轮中,学习者按序支付以打开最多 $n$ 个未知奖励分布的盒子,打开后观察奖励并决定何时停止。学习者的效用为最大观测奖励减去累计开盒成本,目标是最小化后悔,即累积期望效用与最优策略之差。我们提出一种新算法,经 $T$ 轮后实现 $ ilde{O}( oot{nT})$ 后悔率,优于 Agarwal 等人 [2024] 的 $ ilde{O}(n oot{T})$,且接近已知下界(仅对数因子差异)。为更贴近真实应用,我们进一步扩展至自然但具挑战性的上下文线性设定:每个盒子的期望奖励是已知但时变的 $d$ 维上下文的线性函数,噪声分布恒定。我们设计的算法同时学习线性函数与噪声分布,实现 $ ilde{O}(nd oot{T})$ 后悔率。最后,我们证明该技术也适用于在线先知不等式问题,其中学习者必须立即决定是否接受揭示的奖励。在非上下文与上下文设定下,我们的方法均实现类似改进与后悔界。
原文摘要 · Abstract (English)
We study the Pandora's Box problem in an online learning setting with semi-bandit feedback. In each round, the learner sequentially pays to open up to $n$ boxes with unknown reward distributions, observes rewards upon opening, and decides when to stop. The utility of the learner is the maximum observed reward minus the cumulative cost of opened boxes, and the goal is to minimize regret defined as the gap between the cumulative expected utility and that of the optimal policy. We propose a new algorithm that achieves $\widetilde{O}(\sqrt{nT})$ regret after $T$ rounds, which improves the $\widetilde{O}(n\sqrt{T})$ bound of Agarwal et al. [2024] and matches the known lower bound up to logarithmic factors. To better capture real-life applications, we then extend our results to a natural but challenging contextual linear setting, where each box's expected reward is linear in some known but time-varying $d$-dimensional context and the noise distribution is fixed over time. We design an algorithm that learns both the linear function and the noise distributions, achieving $\widetilde{O}(nd\sqrt{T})$ regret. Finally, we show that our techniques also apply to the online Prophet Inequality problem, where the learner must decide immediately whether or not to accept a revealed reward. In both non-contextual and contextual settings, our approach achieves similar improvements and regret bounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。