在内存和批次都受限时,如何高效做多臂赌博机决策?
Few Batches or Little Memory, But Not Both: Simultaneous Space and Adaptivity Constraints in Stochastic Bandits
- 同时限制内存和交互批次,导致算法性能显著下降
- 达到近最优误差需至少 Ω(K/W) 批次,内存越少批次数越多
- 提出新算法,内存与批次可动态平衡,适合资源受限场景
我们研究在空间和自适应性双重约束下的随机多臂赌博机问题:学习者仅能进行 $B$ 个批次的交互,且仅有 $W$ 位持久内存。已有研究表明,单独限制任一条件时,仍可实现近最优误差 $ ilde{O}( oot{KT})$。但当两者同时受限时,性能大幅下降。我们证明,任何 $W$ 位内存的算法至少需要 $Ω(K/W)$ 批次才能达到该误差水平,即使允许完全自适应。特别地,对任意 $ε>0$,$O(K^{1−ε})$ 的批次数被 $O( ext{log}T)$ 内存排除。证明基于信息瓶颈思想:近最优误差要求获得 $Ω(K)$ 位关于优质臂集合的信息,而 $B$ 批次与 $W$ 位内存最多提供 $O(BW)$ 位信息。关键工具是局部变化测度引理,给出概率级探索保证,具有独立价值。此外,我们设计了新算法,在 $W$ 满足 $Ω( ext{log}T) ≤ W ≤ O(K ext{log}T)$ 时,使用不超过 $W$ 位内存与 $ ilde{O}(K/W)$ 批次,实现 $ ilde{O}( oot{KT})$ 误差,几乎逼近下界。
原文摘要 · Abstract (English)
We study stochastic multi-armed bandits under simultaneous constraints on space and adaptivity: the learner interacts with the environment in $B$ batches and has only $W$ bits of persistent memory. Prior work shows that each constraint alone is surprisingly mild: near-minimax regret $\widetilde{O}(\sqrt{KT})$ is achievable with $O(\log T)$ bits of memory under fully adaptive interaction, and with a $K$-independent $O(\log\log T)$-type number of batches when memory is unrestricted. We show that this picture breaks down in the simultaneously constrained regime. We prove that any algorithm with a $W$-bit memory constraint must use at least $Ω(K/W)$ batches to achieve near-minimax regret $\widetilde{O}(\sqrt{KT})$, even under adaptive grids. In particular, logarithmic memory rules out $O(K^{1-\varepsilon})$ batch complexity. Our proof is based on an information bottleneck. We show that near-minimax regret forces the learner to acquire $Ω(K)$ bits of information about the hidden set of good arms under a suitable hard prior, whereas an algorithm with $B$ batches and $W$ bits of memory allows only $O(BW)$ bits of information. A key ingredient is a localized change-of-measure lemma that yields probability-level arm exploration guarantees, which is of independent interest. We also give an algorithm that, for any bit budget $W$ with $Ω(\log T) \le W \le O(K\log T)$, uses at most $W$ bits of memory and $\widetilde{O}(K/W)$ batches while achieving regret $\widetilde{O}(\sqrt{KT})$, nearly matching our lower bound up to polylogarithmic factors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。