提出新基准,让预算受限的在线学习实现可接受的误差。
A New Benchmark for Online Learning with Budget-Balancing Constraints
- 用地球移动距离衡量策略差异,构建更合理的比较基准。
- 在时间窗口内分段调整策略,实现约 $\tilde{O}(T/\sqrt{w}+\sqrt{wT})$ 的误差。
- 证明该基准是必要条件,适合广告竞价等真实场景研究。
对抗性带约束的多臂赌博机(BwK)问题中,学习者每轮选择动作并观测其奖励与成本,目标是在满足预算的前提下最大化总奖励。传统基准为满足预算期望的最佳固定动作分布,但因“花光或保留”困境,该问题对所有实例均无法实现无遗憾学习。本文提出一种基于地球移动距离(EMD)的新基准,证明只要策略支出模式与最优子节奏支出模式的EMD在 $o(T^2)$ 内,即可实现次线性遗憾。作为特例,针对“按窗口调速”基准(将时间划分为大小为 $w$ 的不相交窗口,每个窗口可独立选择动作分布且受节奏预算约束),本算法获得 $ ilde{O}(T/\sqrt{w}+\sqrt{wT})$ 的遗憾上界,并给出匹配下界,证明其最优性。进一步表明EMD条件对获得次线性遗憾至关重要。
原文摘要 · Abstract (English)
The adversarial Bandit with Knapsack problem is a multi-armed bandits problem with budget constraints and adversarial rewards and costs. In each round, a learner selects an action to take and observes the reward and cost of the selected action. The goal is to maximize the sum of rewards while satisfying the budget constraint. The classical benchmark to compare against is the best fixed distribution over actions that satisfies the budget constraint in expectation. Unlike its stochastic counterpart, where rewards and costs are drawn from some fixed distribution (Badanidiyuru et al., 2018), the adversarial BwK problem does not admit a no-regret algorithm for every problem instance due to the "spend-or-save" dilemma (Immorlica et al., 2022). A key problem left open by existing works is whether there exists a weaker but still meaningful benchmark to compare against such that no-regret learning is still possible. In this work, we present a new benchmark to compare against, motivated both by real-world applications such as autobidding and by its underlying mathematical structure. The benchmark is based on the Earth Mover's Distance (EMD), and we show that sublinear regret is attainable against any strategy whose spending pattern is within EMD $o(T^2)$ of any sub-pacing spending pattern. As a special case, we obtain results against the "pacing over windows" benchmark, where we partition time into disjoint windows of size $w$ and allow the benchmark strategies to choose a different distribution over actions for each window while satisfying a pacing budget constraint. Against this benchmark, our algorithm obtains a regret bound of $\tilde{O}(T/\sqrt{w}+\sqrt{wT})$. We also show a matching lower bound, proving the optimality of our algorithm in this important special case. In addition, we provide further evidence of the necessity of the EMD condition for obtaining a sublinear regret.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。