细粒度预训练让模型学得更全面，提升复杂样本的泛化能力。
Why Fine-grained Labels in Pretraining Benefit Generalization?
- 构建分层多视角框架，约束输入数据分布
- 细粒度预训练能学习罕见特征，提升困难样本准确率
- 适合研究模型泛化与预训练策略的读者
近期研究表明，用细粒度标签数据预训练深层神经网络，再在粗粒度标签数据上微调，通常比直接用粗粒度数据预训练更能提升下游任务的泛化性能。尽管已有大量实证支持，但其理论依据仍不明确。本文提出‘分层多视角’结构来约束输入数据分布，在该框架下证明：1）粗粒度预训练仅使网络充分学习共性特征；2）细粒度预训练则有助于网络同时学习罕见特征与共性特征，从而提升对困难下游测试样本的准确率。
原文摘要 · Abstract (English)
Recent studies show that pretraining a deep neural network with fine-grained labeled data, followed by fine-tuning on coarse-labeled data for downstream tasks, often yields better generalization than pretraining with coarse-labeled data. While there is ample empirical evidence supporting this, the theoretical justification remains an open problem. This paper addresses this gap by introducing a "hierarchical multi-view" structure to confine the input data distribution. Under this framework, we prove that: 1) coarse-grained pretraining only allows a neural network to learn the common features well, while 2) fine-grained pretraining helps the network learn the rare features in addition to the common ones, leading to improved accuracy on hard downstream test samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。