提出统一框架,让模型一次训练即可适配任意下游任务和损失函数。
Panprediction: Optimal Predictions for Any Downstream Task and Loss
- 构建了可同时应对无限任务与损失的泛预测理论框架
- 确定性与随机性预测器分别仅需约1/ε³和1/ε²样本
- 首次实现跨任务与损失的高效学习,适合需要通用模型的场景
传统监督学习聚焦于在固定分布和损失下训练模型。而新兴范式则将模型训练视为从数据中提取足够信息,使模型能用于最小化多种损失下的多个下游任务。本文提出名为‘泛预测’(panprediction)的数学框架,并研究其统计复杂度。该框架广义化了‘全预测’(omniprediction),并位于‘多组学习’(multi-group learning)之前,前者关注对多种损失的泛化,后者关注对多种任务的泛化,但两者均未兼顾。具体地,本文设计出确定性和随机性泛预测器算法,分别只需 ilde{O}(1/ε^3)和 ilde{O}(1/ε^2)个样本。结果表明,在温和假设下,同时最小化无限多个损失和任务的统计难度,可等同于最小化单一损失与任务。此外,我们改进了确定性全预测的样本复杂度,提升因子1/ε,且匹配了全预测与多组学习的所有已知复杂度界。核心技术是将泛预测近乎无损地转化为一种统计高效的校准概念——分步校准(step calibration)。
原文摘要 · Abstract (English)
Supervised learning is classically formulated as training a model to minimize a fixed loss function over a fixed distribution, or task. However, an emerging paradigm instead views model training as extracting enough information from data so that the model can be used to minimize many losses on many downstream tasks. We formalize a mathematical framework for this paradigm, which we call panprediction, and study its statistical complexity. Formally, panprediction generalizes omniprediction and sits upstream from multi-group learning, which respectively focus on predictions that generalize to many downstream losses or many downstream tasks, but not both. Concretely, we design algorithms that learn deterministic and randomized panpredictors with $\tilde{O}(1/\varepsilon^3)$ and $\tilde{O}(1/\varepsilon^2)$ samples, respectively. Our results demonstrate that under mild assumptions, simultaneously minimizing infinitely many losses on infinitely many tasks can be as statistically easy as minimizing one loss on one task. Along the way, we improve the best known sample complexity guarantee of deterministic omniprediction by a factor of $1/\varepsilon$, and match all other known sample complexity guarantees of omniprediction and multi-group learning. Our key technical ingredient is a nearly lossless reduction from panprediction to a statistically efficient notion of calibration, called step calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。