利用黑箱模型提升小样本回归，实现安全纠错并发现性能拐点。
Black-Box Assisted Regression: Phase Transitions and Minimax Optimality
- 基于黑箱预测构建残差修正模型，初始等同于黑箱，可自动回退避免负迁移。
- 在误差半径δ与样本数n的临界值δ_c(n)处出现性能相变，最优风险为min{δ², n^{-2β/(2β+d)}}。
- 适用于高风险场景中依赖预训练模型但数据稀缺的机器学习任务。
基础模型常被用作下游任务的固定黑箱预测器，尤其在标注数据有限时，但其预测可能有偏且不可盲目信赖。本文研究黑箱辅助非参数回归：学习者观察到带标签样本，并可查询一个固定预测器 $f_0$，目标函数 $f^*$ 在 $L_2(P_X)$ 范围内与 $f_0$ 的距离不超过未知半径 $δ$。我们给出有限样本下的极小极大特性，揭示在 $δ_c(n) acksim n^{-β/(2β+d)}$ 处存在相变，主导风险为 $ ext{min}igackslash{δ^2, n^{-2β/(2β+d)}igackslash}$。随后分析一种安全残差估计器：它在 $f_0$ 周围学习修正项，初始化残差头为零（即初始预测等于 $f_0$），并通过留出集选择机制，在验证数据不支持修正时回退至 $f_0$，从而确保‘安全’——避免性能劣于黑箱本身。该估计器达到极小极大主导项，仅额外承担验证选择成本。合成回归实验验证了预测的相变现象；在 CIFAR-100 上使用 CLIP、AG News 上使用 Qwen3-8B 的实证表明，相同的残差修正权衡在平方损失之外的实用场景中依然有效。
原文摘要 · Abstract (English)
Foundation models are often used as fixed black-box predictors for downstream tasks with limited labeled data, but their predictions may be biased and unsafe to trust blindly. We study this setting through black-box assisted nonparametric regression: a learner observes labeled samples and can query a fixed predictor $f_0$, while the target $f^*$ is close to $f_0$ in $L_2(P_X)$ up to an unknown radius $δ$. We give a finite-sample minimax characterization showing a phase transition at $δ_c(n) \asymp n^{-β/(2β+d)}$, with leading risk $\min\{δ^2, n^{-2β/(2β+d)}\}$. We then analyze a Safe Residual Estimator: it learns a correction around $f_0$, initializes the residual head at zero so the initial predictor equals $f_0$, and uses holdout selection to revert to $f_0$ when the learned correction is not supported by validation data. Here, "safe" means avoiding negative transfer, i.e., performing worse than the black-box predictor alone. The estimator matches the leading minimax term up to an additive validation-selection cost. Synthetic regression experiments verify the predicted phase transition, while CIFAR-100 with CLIP and AG News with Qwen3-8B provide practice-facing evidence that the same residual-correction tradeoff is useful beyond the formal squared-loss regression setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。