融合黑箱与白箱模型,自动找到最优预测策略组合。
Learning the Pareto Frontier of Predictive Models under Distribution Shift

- 构建统一特征:拼接白箱模型内部表征与黑箱模型输出。
- 在真实数据上表现超越单一方法,尤其在无可靠基线时优势明显。
- 适合跨领域部署中需灵活应对分布偏移的场景。
现代机器学习流水线越来越依赖在下游任务中复用预训练和基础模型。这些模型不仅性能不同,使用方式也各异:部分仅提供黑箱预测,另一些则允许访问内部表征以进行探测或微调。当部署到存在分布偏移的目标域时,零样本应用、微调或直接训练目标专用模型等任一策略均无法始终最优。本文提出前沿学习(Frontier Learning)框架,将涵盖不同训练历史和访问模式的候选模型视为互补信息源而非互斥选项。该框架通过拼接白箱候选模型的内部表征与黑箱候选模型的预测输出,构建统一的目标域特征,并在此基础上使用标注目标数据拟合一个轻量级正则化监督学习器。由于所得假设类包含零样本复用、微调及直接训练等策略作为特例,对前沿学习器的经验风险最小化在训练样本上必然不劣于任何单一基线。我们在模拟实验中评估了多种源-目标兼容性情况,并在两个真实分布偏移场景下验证:DomainNet/VisDA上的视觉域适应,以及基于MIMIC-IV-Notes的重症监护室临床死亡率预测。在所有设置中,前沿学习均匹配或优于最强个体复用策略,且最大提升出现在无单一基线可靠的分布偏移范围内。
原文摘要 · Abstract (English)
Modern machine learning pipelines increasingly rely on reusing pretrained and foundation models across downstream tasks. These pretrained models can differ not only in performance but also in how they can be used: some only provide black-box predictions, while others may permit white-box access to internal representations that can be probed or fine-tuned. When deployed to the target domain in the presence of distribution shift, no single strategy, including zero-shot application, fine-tuning, or directly training a target-specific model, is uniformly the best. In this work, we propose Frontier Learning, a framework that treats a library of candidate models spanning different training histories and access regimes as complementary sources of information rather than mutually exclusive alternatives. Frontier Learning constructs a unified target-domain feature by concatenating internal representations from white-box candidates as well as prediction outputs from black-box candidates, then fits a lightweight, regularized supervised learner on this concatenated representation using labeled target data. Because the resulting hypothesis class contains predictors obtained by zero-shot reuse, fine-tuning, and direct training as special cases, empirical risk minimization over the frontier learner is guaranteed to be no worse, on the training sample, than any individual baseline. We evaluate the framework in simulations spanning varying degrees of source-target compatibility and in two real-world distribution-shift settings: visual domain adaptation on DomainNet/VisDA and clinical mortality prediction across intensive care unit domains using MIMIC-IV-Notes. Across all settings, Frontier Learning matches or outperforms the strongest individual reuse strategy, with the largest gains arising precisely when no single baseline is reliable across the range of shift considered.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。