提出近最优的全能预测算法,可同时适应多种损失函数。
Near-Optimal Algorithms for Omniprediction
- 设计在线学习算法,实现近最优后悔值。
- 离线算法在有限样本下逼近最优泛化误差。
- 适合需要多目标优化的机器学习场景。
全能预测器是能针对假设类 $H$ 内所有损失函数 $L$,同时给出最小化损失的预测函数。本文在在线与离线两种设置下,给出了近最优的学习算法。首先,提出一个基于预言机的在线算法,在任意满足利普希茨条件的损失类 $L \subseteq L_\mathrm{Lip}$ 下,达到 $\tilde O (\sqrt{T \log |H|})$ 的后悔值,该结果与单个损失最小化最优后悔值仅差 $\sqrt{\log(T)}$ 因子。基于此在线算法,我们构建了在线到离线的转换,得到离线学习算法:对于有界变差损失类 $L_\mathrm{BV}$(包含所有凸、利普希茨和正确损失)及任意(可能无限)的 $H$,利用 $m$ 个来自分布 $D$ 的样本和一个(离线)ERM预言机,可输出一个高效的 $(L_{\mathrm{BV}},H,ε(m))$-全能预测器,其中 $ε(m)$ 近似线性地依赖于由 $H$ 经固定数量元素凸组合构造出类的 Rademacher 复杂度。
原文摘要 · Abstract (English)
Omnipredictors are simple prediction functions that encode loss-minimizing predictions with respect to a hypothesis class $H$, simultaneously for every loss function within a class of losses $L$. In this work, we give near-optimal learning algorithms for omniprediction, in both the online and offline settings. To begin, we give an oracle-efficient online learning algorithm that acheives $(L,H)$-omniprediction with $\tilde O (\sqrt{T \log |H|})$ regret for any class of Lipschitz loss functions $L \subseteq L_\mathrm{Lip}$. Quite surprisingly, this regret bound matches the optimal regret for \emph{minimization of a single loss function} (up to a $\sqrt{\log(T)}$ factor). Given this online algorithm, we develop an online-to-offline conversion that achieves near-optimal complexity across a number of measures. In particular, for all bounded loss functions within the class of Bounded Variation losses $L_\mathrm{BV}$ (which include all convex, all Lipschitz, and all proper losses) and any (possibly-infinite) $H$, we obtain an offline learning algorithm that, leveraging an (offline) ERM oracle and $m$ samples from $D$, returns an efficient $(L_{\mathrm{BV}},H,ε(m))$-omnipredictor for $\varepsilon(m)$ scaling near-linearly in the Rademacher complexity of a class derived from $H$ by taking convex combinations of a fixed number of elements of $\mathrm{Th} \circ H$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。