用数据增强的变分目标自动学超参数，省去验证集和耗时调参。
Learning Hyperparameters via a Data-Emphasized Variational Objective
- 基于变分推断的梯度优化方法，直接从训练数据学习超参数。
- 在图像与文本分类任务中，将调参时间从88小时缩短至3小时以内。
- 适合资源有限、需快速调参的模型训练场景，尤其适合迁移学习。
在数据有限的情况下训练大型模型，避免过拟合至关重要。传统网格搜索或智能搜索方法需要为每个候选超参数进行独立训练，且需划分验证集，减少了可用于训练的数据量。本文研究通过贝叶斯变分方法中的证据下界（ELBO）实现超参数的梯度学习，无需任何验证集。我们关注模型过参数化、近似后验设为各向同性高斯分布以保证可计算性的场景。然而，发现标准ELBO会优先匹配先验，导致严重欠拟合。因此，我们提出数据增强型ELBO，提高似然项权重但不调整先验项。在图像与文本分类的贝叶斯迁移学习中，该方法将以往超过88小时的网格搜索压缩至3小时以内，同时保持相当的精度。此外，还展示了该方法能高效准确地逼近具有可学习长度尺度核的高斯过程。
原文摘要 · Abstract (English)
When training large models on limited data, avoiding overfitting is paramount. Common grid search or smarter search methods rely on expensive separate runs for each candidate hyperparameter, while carving out a validation set that reduces available training data. In this paper, we study gradient-based learning of hyperparameters via the evidence lower bound (ELBO) objective from Bayesian variational methods. This avoids the need for any validation set. We focus on scenarios where the model is over-parameterized for flexibility and the approximate posterior is chosen to be Gaussian with isotropic covariance for tractability, even though it cannot match the true posterior. In such scenarios, we find the ELBO prioritizes posteriors that match the prior, leading to severe underfitting. Instead, we recommend a data-emphasized ELBO that upweights the likelihood but not the prior. In Bayesian transfer learning of image and text classifiers, our method reduces the 88+ hour grid search of past work to under 3 hours while delivering comparable accuracy. We further demonstrate how our approach enables efficient yet accurate approximations of Gaussian processes with learnable lengthscale kernels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。