用贝叶斯方法选预训练模型最优检查点，无需下游数据。
A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints
- 基于下游自由能的贝叶斯准则，衡量检查点对下游任务的适应潜力。
- 该准则在无下游数据情况下仍有效，与微调性能高度相关。
- 适合做预训练模型筛选的研究者和工程师参考。
近年来，以BERT、GPT、T5和视觉变换器为代表的基座模型推动了人工智能发展。这些模型先在大规模多样数据上预训练，再针对具体下游任务微调，通常只需少量数据。然而，这种“预训练-微调”范式成功背后的机制仍未充分探索，尤其是哪些预训练检查点更利于下游适应。本文提出一种贝叶斯模型选择准则——下游自由能，通过测量下游任务附近优良参数的集中程度来量化检查点的可适应性。我们证明该准则可在不访问下游数据或不了解下游任务的情况下有效实现。此外，实证结果表明该准则与微调性能有可靠相关性，为预测模型适应性提供了理论依据。
原文摘要 · Abstract (English)
Recent advances in artificial intelligence have been fueled by the development of foundation models such as BERT, GPT, T5, and Vision Transformers. These models are first pretrained on vast and diverse datasets and then adapted to specific downstream tasks, often with significantly less data. However, the mechanisms behind the success of this ubiquitous pretrain-then-adapt paradigm remain underexplored, particularly the characteristics of pretraining checkpoints that enhance downstream adaptation. We introduce a Bayesian model selection criterion, called the downstream free energy, which quantifies a checkpoint's adaptability by measuring the concentration of nearby favorable parameters for the downstream task. We demonstrate that this Bayesian model selection criterion can be effectively implemented without access to the downstream data or prior knowledge of the downstream task. Furthermore, we provide empirical evidence that the criterion reliably correlates with improved finetuning performance, offering a principled approach to predicting model adaptability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。