arXiv:2412.19085cs.LG2024-12AAAI被引 13

通过频谱成分分布评估预训练模型,提升迁移学习选型效率。

Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components

  • 用特征奇异值分解分析频谱成分转移能力差异
  • 聚焦高转移性成分占比,筛选最优预训练模型
  • 适用于分类与回归任务,可替代耗时微调

针对迁移学习中的预训练模型选型问题,现有方法多关注整体特征内在属性或对目标标签的拟合程度。本文提出一种新视角——通过特征奇异值分解分析频谱成分分布(DISCO),发现不同成分具有不同转移能力,对微调性能贡献各异。基于此,提出一种基于频谱成分分布的评估方法,衡量各成分对应奇异值的比例;将特征集中在更易转移成分上的模型视为更优选择。进一步结合下游数据标签,优化各成分转移性估计,构建最终评估准则。该方法灵活,适用于分类与回归任务。在三个基准和两类任务(图像分类、目标检测)上进行综合实验,结果表明其在从模型库中选择合适预训练模型方面达到当前最佳性能。

原文摘要 · Abstract (English)

Pre-trained model assessment for transfer learning aims to identify the optimal candidate for the downstream tasks from a model hub, without the need of time-consuming fine-tuning. Existing advanced works mainly focus on analyzing the intrinsic characteristics of the entire features extracted by each pre-trained model or how well such features fit the target labels. This paper proposes a novel perspective for pre-trained model assessment through the Distribution of Spectral Components (DISCO). Through singular value decomposition of features extracted from pre-trained models, we investigate different spectral components and observe that they possess distinct transferability, contributing diversely to the fine-tuning performance. Inspired by this, we propose an assessment method based on the distribution of spectral components which measures the proportions of their corresponding singular values. Pre-trained models with features concentrating on more transferable components are regarded as better choices for transfer learning. We further leverage the labels of downstream data to better estimate the transferability of each spectral component and derive the final assessment criterion. Our proposed method is flexible and can be applied to both classification and regression tasks. We conducted comprehensive experiments across three benchmarks and two tasks including image classification and object detection, demonstrating that our method achieves state-of-the-art performance in choosing proper pre-trained models from the model hub for transfer learning.

迁移学习模型评估特征分析频谱分布

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。