从最优传输角度证明迁移学习能显著降低样本需求。
Sample Complexity of Transfer Learning: An Optimal Transport Approach
- 用最优传输理论分析迁移学习的样本效率
- 当数据维度大于3时，迁移学习样本复杂度为O(m^(-(α+1)/d))
- 适合小样本、复杂模型（如非光滑激活函数）场景使用
迁移学习是大型语言模型和生成式AI等复杂结构模型的关键技术。其核心是从已解决的源任务中迁移知识以加速新目标任务的学习，尤其在目标任务训练数据量m较小的情况下优势明显。本文从最优传输视角严格分析迁移学习的样本效率：当数据维度d > 3时，迁移学习的样本复杂度为O(m^(-(α+1)/d))，其中α表示数据分布的平滑性；而直接学习的复杂度为O(m^(-p/d))，p为最优目标模型的平滑性。理论表明，在目标任务优化的是非平滑复杂模型（如含非光滑激活函数的深度网络）时，迁移学习具有更优的样本效率。以图像分类为例，数值实验验证了在数据稀缺场景下，迁移学习可显著提升模型性能。
原文摘要 · Abstract (English)
Transfer learning is an essential technique for many machine learning/AI models of complex structures such as large language models and generative AI. The essence of transfer learning is to leverage knowledge from resolved source tasks for a new target task, especially when the sample size $m$ of the training data for the latter is low. In this work, we rigorously analyze the potential benefit of transfer learning in terms of sample efficiency. Specifically, taking an optimal transport viewpoint of transfer learning, we find that when the data dimension $d$ is higher than $3$, the sample complexity for transfer learning is $O(m^{-(α+1)/d})$, with $α$ indicating the smoothness of the data distribution, as opposed to the $O(m^{-p/d})$ sample complexity for direct learning with $p$ indicating the smoothness of the optimal target model. Our finding theoretically supports a better sample efficiency for transfer learning, when the target task is optimizing over a family of not-so-smooth models (i.e., highly complex networks with the possible use of non-smooth activation functions). Using image classification as an example, we numerically demonstrate the sample efficiency for transfer learning, that is, in the data hungry regime, the model performance can be significantly improved by transfer learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。