多预训练模型迁移学习中，过参数化导致偏差，用修正因子可提升效果。
Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing
- 融合多个过参数化预训练模型，通过距离惩罚优化目标任务。
- 使用足够多预训练模型能降低测试误差，但存在过参数化偏差。
- 提出乘法修正因子，有效缓解偏差，适合高维线性建模场景。
我们研究了基于多个最小二乘预训练模型的线性回归迁移学习任务。将目标学习问题建模为在目标数据集上最小化平方误差，并对学习模型与预训练模型之间的距离施加惩罚。我们分析推导了目标模型的测试误差表达式，并进行了相应的实证评估。结果揭示了使用更多预训练模型为何能提升迁移学习性能：若预训练模型为过参数化，使用足够多的模型对有益于迁移学习。然而，过参数化可能引入偏差，即最小ℓ₂范数解被限制在高维参数空间中由训练样本张成的小子空间内。为此，我们提出一种简单的乘法修正因子方法，可减少过参数化偏差，从而更有效地利用多个预训练模型来学习目标预测器。
原文摘要 · Abstract (English)
We study transfer learning for a linear regression task using several least-squares pretrained models that can be overparameterized. We formulate the target learning task as optimization that minimizes squared errors on the target dataset with penalty on the distance of the learned model from the pretrained models. We analytically formulate the test error of the learned target model and provide the corresponding empirical evaluations. Our results elucidate when using more pretrained models can improve transfer learning. Specifically, if the pretrained models are overparameterized, using sufficiently many of them is important for beneficial transfer learning. However, the learning may be compromised by overparameterization bias of pretrained models, i.e., the minimum $\ell_2$-norm solution's restriction to a small subspace spanned by the training examples in the high-dimensional parameter space. We propose a simple debiasing via multiplicative correction factor that can reduce the overparameterization bias and leverage more pretrained models to learn a target predictor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。