解析参数迁移中哪些部分有用,为何有时迁移反而降低效果。
Towards Understanding Feature Learning in Parameter Transfer
- 用理论分析证明部分参数迁移的条件与机制。
- 发现迁移可能降低准确率,首次从理论上证明负迁移存在。
- 适用于研究迁移学习机理的科研人员和算法设计者。
参数迁移是迁移学习的核心范式,通过在上游和下游模型间共享参数实现知识复用。然而,当仅迁移上游模型的部分参数时,目前尚缺乏对这种局部参数复用何时有益及其有效性影响因素的理论理解。为此,本文分析了上游和下游模型均为ReLU卷积神经网络(CNNs)的情形,在此理论框架下,刻画了继承参数如何承载通用知识,并识别出放大其对目标任务积极影响的关键因素。此外,我们的分析揭示了为何在某些情况下,参数迁移会导致目标任务测试准确率低于从零开始训练新模型的情况。据我们所知,本理论是首个对参数迁移进行动态分析的工作,也是首个从理论上证明负迁移存在的研究。通过数值实验和真实数据实验,验证了理论结果的正确性。
原文摘要 · Abstract (English)
Parameter transfer is a central paradigm in transfer learning, enabling knowledge reuse across tasks and domains by sharing model parameters between upstream and downstream models. However, when only a subset of parameters from the upstream model is transferred to the downstream model, there remains a lack of theoretical understanding of the conditions under which such partial parameter reuse is beneficial and of the factors that govern its effectiveness. To address this gap, we analyze a setting in which both the upstream and downstream models are ReLU convolutional neural networks (CNNs). Within this theoretical framework, we characterize how the inherited parameters act as carriers of universal knowledge and identify key factors that amplify their beneficial impact on the target task. Furthermore, our analysis provides insight into why, in certain cases, transferring parameters can lead to lower test accuracy on the target task than training a new model from scratch. To our best knowledge, our theory is the first to provide a dynamic analysis for parameter transfer and also the first to prove the existence of negative transfer theoretically. Numerical experiments and real-world data experiments are conducted to empirically validate our theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。