arXiv:2605.08287cs.LGcs.AI2026-05被引 1

带最优动作查询的老虎机问题,揭示了查询如何影响学习效率

Multi-Armed Bandits With Best-Action Queries

  • 引入最优动作查询机制,探索其在仅观测所选动作奖励时的效果
  • 在独立同分布奖励下,可实现近似最优的最小化后悔值
  • 首次完整刻画该设置下查询带来的理论收益边界

我们研究了引入最优动作查询的多臂老虎机问题,即学习者可额外向一个预言机查询当前轮次中表现最好的动作。此前,Russo 等人 [2024] 在全反馈模型中证明:在随机与对抗性环境中,$k$ 次查询可将最优 $ ilde{ m O}( oot T o)$ 的后悔率降至 $ ilde{ m O}( ext{min}egin{Bmatrix}T/k, oot T oegin{Bmatrix})$。但这一结果是否适用于更现实的仅观测所选动作奖励的带反馈模型,仍为开放问题。本文完全解决此问题:当奖励在各动作间存在相关性时,任何算法的后悔至少为 $Ω( oot T-k o)$,该下界也适用于对抗环境。在奖励独立同分布的随机情形下,我们构造出能达到 $ ilde{ m O}( ext{min}egin{Bmatrix}T/k, oot T-k oegin{Bmatrix})$ 后悔的算法,并建立匹配的下界(对数因子内)。综上,本文完整刻画了带反馈模型中最佳动作查询的理论优势。

原文摘要 · Abstract (English)

We study \emph{multi-armed bandits} (MABs) augmented with \emph{best-action queries}, in which the learner may additionally query an oracle that reveals the best arm in the current round. This setting was recently characterized by Russo et al. [2024] in the \emph{full-feedback} model, where the learner observes the rewards of all arms after each round. They show that, in both \emph{stochastic} and \emph{adversarial} environments, $k$ best-action queries reduce the optimal $\widetilde{\mathcal{O}}(\sqrt{T})$ regret to $\widetilde{\mathcal{O}}(\min\{T/k,\sqrt{T}\})$. Whether this improvement extends to the more realistic \emph{bandit-feedback} model -- where the learner observes only the reward of the played arm -- was left as an open problem. We fully resolve this question. When rewards are stochastic but correlated among arms, we show that the full-feedback result does not carry over: any algorithm must incur regret at least $Ω(\sqrt{T-k})$. This lower bound directly extends to adversarial environments. On the positive side, we show that $\widetilde{\mathcal{O}}(\min\{T/k,\sqrt{T-k}\})$ regret is still achievable when rewards are stochastic and i.i.d., and establish a matching lower bound, up to logarithmic factors. Together, these results provide a complete characterization of the benefits of \emph{best-action queries} in the \emph{bandit-feedback} model.

强化学习在线学习优化理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。