首次完整刻画含噪声随机老虎机模型的可学习性与最优查询复杂度。
A Complete Characterization of Learnability for Stochastic Noisy Bandits
- 基于噪声模型构建可学习性判定框架,突破以往不可判定困境。
- 揭示最优查询复杂度存在多种可能取值,且需自适应策略才能达到。
- 提出新变体决策-估计系数,统一刻画该问题的可学习性边界。
我们研究已知函数类 𝒫 中未知奖励函数 𝑓^* 的随机噪声老虎机问题。给定模型类 𝒫,每个模型 𝑀 将动作 π 映射为奖励分布 𝑀(π),其期望奖励函数定义为 𝑓^𝑀(π) = ℰ_{𝑟∼𝑀(π)}[𝑟]。在每轮中,算法选择一个动作 π 并观测来自 𝑀(π) 的奖励样本。若已知 𝒫 且真实模型 𝑀 ∈ 𝒫,目标是在有限轮数内以高概率识别出近似最大均值奖励的动作 𝜋̂。若此可行,则称 𝒫 可学习。此前工作表明某些确定性奖励模型类的可学习性不可判定,但其不包含噪声。本文首次正面回答:对于任意噪声模型类,可学习性是可判定的。我们给出完整的可学习性刻画、所有可能的最优查询复杂度谱,并证明自适应策略有时是实现最优复杂度所必需的。此外,我们重新审视决策-估计系数(DEC),提出其新变体,同样能刻画此类问题的可学习性。
原文摘要 · Abstract (English)
We study the stochastic noisy bandit problem with an unknown reward function $f^*$ in a known function class $\mathcal{F}$. Formally, a model $M$ maps arms $π$ to a probability distribution $M(π)$ of reward. A model class $\mathcal{M}$ is a collection of models. For each model $M$, define its mean reward function $f^M(π)=\mathbb{E}_{r \sim M(π)}[r]$. In the bandit learning problem, we proceed in rounds, pulling one arm $π$ each round and observing a reward sampled from $M(π)$. With knowledge of $\mathcal{M}$, supposing that the true model $M\in \mathcal{M}$, the objective is to identify an arm $\hatπ$ of near-maximal mean reward $f^M(\hatπ)$ with high probability in a bounded number of rounds. If this is possible, then the model class is said to be learnable. Importantly, a result of \cite{hanneke2023bandit} shows there exist model classes for which learnability is undecidable. However, the model class they consider features deterministic rewards, and they raise the question of whether learnability is decidable for classes containing sufficiently noisy models. For the first time, we answer this question in the positive by giving a complete characterization of learnability for model classes with arbitrary noise. In addition to that, we also describe the full spectrum of possible optimal query complexities. Further, we prove adaptivity is sometimes necessary to achieve the optimal query complexity. Last, we revisit an important complexity measure for interactive decision making, the Decision-Estimation-Coefficient \citep{foster2021statistical,foster2023tight}, and propose a new variant of the DEC which also characterizes learnability in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。