用数据加权的变分目标直接学习正则化强度，省去调参和验证集
Learning the Regularization Strength for Deep Fine-Tuning via a Data-Emphasized Variational Objective
- 基于变分下界改进目标函数，直接在全训练集上学习正则化强度
- 在多个图像分类数据集上达到与现有方法相当的准确率，计算耗时显著降低
- 适合追求高效微调、不想手动调参的研究者和工程师
许多流行的迁移学习方法依赖网格搜索来选择控制过拟合的正则化超参数。该方法存在三大缺陷：计算成本高、需预留验证集减少训练数据、需人工指定候选值。本文提出一种替代方案：通过基于证据下界（ELBo）的变分模型选择技术，在完整训练集上直接学习正则化超参数。针对含数百万参数的深度神经网络，我们推荐一种改进的ELBo，增强数据似然的影响权重，同时保持对证据的有效下界。该方法克服了网格搜索的所有弊端。我们在多个图像分类数据集上验证了有效性，获得与现有方法相当的泛化准确率，且所需计算时间大幅减少。
原文摘要 · Abstract (English)
A number of popular transfer learning methods rely on grid search to select regularization hyperparameters that control over-fitting. This grid search requirement has several key disadvantages: the search is computationally expensive, requires carving out a validation set that reduces the size of available data for model training, and requires practitioners to specify candidate values. In this paper, we propose an alternative to grid search: directly learning regularization hyperparameters on the full training set via model selection techniques based on the evidence lower bound ("ELBo") objective from variational methods. For deep neural networks with millions of parameters, we specifically recommend a modified ELBo that upweights the influence of the data likelihood relative to the prior while remaining a valid bound on the evidence for Bayesian model selection. Our proposed technique overcomes all three disadvantages of grid search. We demonstrate effectiveness on image classification tasks on several datasets, yielding heldout accuracy comparable to existing approaches with far less compute time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。