提出LGD算法，用贝叶斯最优解指导梯度下降超参调优，实现小样本线性回归的可靠泛化。
Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates
- 用朗之万更新近似损失函数后验均值，实现超参自适应调优
- 证明在凸回归任务中存在使模型达贝叶斯最优的超参配置
- 首次将弹性网络的泛化界扩展至多维超参，适用于小样本学习
我们从超参数调优视角研究回归问题中的学习元学习。提出朗之万梯度下降算法（LGD），通过损失函数与正则项定义的后验分布均值进行近似。证明存在一个最优超参配置，使LG D算法在平方损失下达到贝叶斯最优解。进一步，在数据驱动设定下，研究从一组任务中元学习LGD最优超参的泛化保证。在参数维度d和超参维度h的条件下，我们得到伪维数上界为O(dh)，包含对数项，前提是对LGD的温和假设。该结果与之前仅支持两超参的弹性网络边界在维度依赖上一致，并将其推广至凸损失下的回归任务。最后，通过合成数据集上的实验证明了LGD及元学习流程在小样本线性回归中的有效性。
原文摘要 · Abstract (English)
We study learning to learn for regression problems through the lens of hyperparameter tuning. We propose the Langevin Gradient Descent Algorithm (LGD), which approximates the mean of the posterior distribution defined by the loss function and regularizer of a convex regression task. We prove the existence of an optimal hyperparameter configuration for which the LGD algorithm achieves the Bayes' optimal solution for squared loss. Subsequently, we study generalization guarantees on meta-learning optimal hyperparameters for the LGD algorithm from a given set of tasks in the data-driven setting. For a number of parameters $d$ and hyperparameter dimension $h$, we show a pseudo-dimension bound of $O(dh)$, upto logarithmic terms under mild assumptions on LGD. This matches the dimensional dependence of the bounds obtained in prior work for the elastic net, which only allows for $h=2$ hyperparameters, and extends their bounds to regression on convex loss. Finally, we show empirical evidence of the success of LGD and the meta-learning procedure for few-shot learning on linear regression using a few synthetically created datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。