利用离线数据降低线性多臂问题的在线后悔,显著提升推荐系统等场景效率。
Regret minimization in Linear Bandits with offline data via extended D-optimal exploration
- 通过扩展D-最优设计融合离线数据,在探索阶段优化信息获取。
- 在线后悔上界为$ ilde{O}( oot ext{eff} T "log(|A|T)+d^2$,$ ext{eff}$反映离线数据质量。
- 首次给出依赖离线数据质量的下界,证明算法在不同数据条件下最优。
我们研究在可访问先验观测(离线数据)的线性多臂问题中实现在线后悔最小化。此类问题在推荐系统、在线广告等领域有广泛应用。本文提出离线-在线分段淘汰算法(OOPE),通过在每个探索阶段引入扩展D-最优设计,有效利用离线数据,显著降低在线后悔。算法的在线后悔上界为$ ilde{O}( oot ext{eff} T "log(| ext{A}|T)+d^2$,其中$ ext{eff} \\(leq d)$为有效问题维度,衡量离线数据中未充分探索方向的数量,其值取决于离线数据格拉姆矩阵的特征谱$(λ_k)_{k \in [d]}$。特征谱是离线数据质量的量化指标:当离线数据不足($ ext{eff} \approx d$)时,回归纯在线设置的已知界限;当离线数据丰富($ ext{T}_{ ext{off}} >> T$)且良好探索($ ext{eff} = o(1)$)时,在线后悔大幅下降。此外,本文首次给出了该设置下显式依赖离线数据质量的极小极大后悔下界,证明了算法在离线数据良好或欠探索情形下的最优性。最后,通过弗兰克-沃尔夫近似扩展最优设计,将$O(d^2)$项改进为$O\left(\frac{d^2}{\text{eff}} \min \{\text{eff},1\} \right)$,在高维且离线数据质量中等($ ext{eff} = Ω(1)$)时提升显著。
原文摘要 · Abstract (English)
We consider the problem of online regret minimization in linear bandits with access to prior observations (offline data) from the underlying bandit model. There are numerous applications where extensive offline data is often available, such as in recommendation systems, online advertising. Consequently, this problem has been studied intensively in recent literature. Our algorithm, Offline-Online Phased Elimination (OOPE), effectively incorporates the offline data to substantially reduce the online regret compared to prior work. To leverage offline information prudently, OOPE uses an extended D-optimal design within each exploration phase. OOPE achieves an online regret is $\tilde{O}(\sqrt{\deff T \log \left(|\mathcal{A}|T\right)}+d^2)$. $\deff \leq d)$ is the effective problem dimension which measures the number of poorly explored directions in offline data and depends on the eigen-spectrum $(λ_k)_{k \in [d]}$ of the Gram matrix of the offline data. The eigen-spectrum $(λ_k)_{k \in [d]}$ is a quantitative measure of the \emph{quality} of offline data. If the offline data is poorly explored ($\deff \approx d$), we recover the established regret bounds for purely online setting while, when offline data is abundant ($\Toff >> T$) and well-explored ($\deff = o(1) $), the online regret reduces substantially. Additionally, we provide the first known minimax regret lower bounds in this setting that depend explicitly on the quality of the offline data. These lower bounds establish the optimality of our algorithm in regimes where offline data is either well-explored or poorly explored. Finally, by using a Frank-Wolfe approximation to the extended optimal design we further improve the $O(d^{2})$ term to $O\left(\frac{d^{2}}{\deff} \min \{ \deff,1\} \right)$, which can be substantial in high dimensions with moderate quality of offline data $\deff = Ω(1)$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。