arXiv:2410.10230stat.MLcs.LG2024-10NeurIPS被引 1

用代理目标优化替代直接优化风险,提升学习算法的效率与泛化能力。

Learning via Surrogate PAC-Bayes

  • 用低维函数空间投影替代原经验风险,实现高效迭代优化
  • 理论证明代理优化可等价于原始泛化界优化
  • 应用于元学习框架,获得可解析的元梯度表达式

PAC-Bayes 学习为研究学习算法泛化能力及基于泛化界构造新算法提供了完整框架。然而,直接优化泛化界在计算上可能不可行。为此,本文提出一种基于代理目标的迭代学习策略:将泛化界中的经验风险替换为可构造的低维函数空间上的投影,该投影可更高效地查询。我们首先建立理论结果,证明迭代优化代理目标等价于优化原始泛化界;其次将该策略应用于元学习框架,提出一个具有闭式表达的元目标,可导出显式元梯度;最后通过受工业生化问题启发的数值实验验证方法有效性。

原文摘要 · Abstract (English)

PAC-Bayes learning is a comprehensive setting for (i) studying the generalisation ability of learning algorithms and (ii) deriving new learning algorithms by optimising a generalisation bound. However, optimising generalisation bounds might not always be viable for tractable or computational reasons, or both. For example, iteratively querying the empirical risk might prove computationally expensive. In response, we introduce a novel principled strategy for building an iterative learning algorithm via the optimisation of a sequence of surrogate training objectives, inherited from PAC-Bayes generalisation bounds. The key argument is to replace the empirical risk (seen as a function of hypotheses) in the generalisation bound by its projection onto a constructible low dimensional functional space: these projections can be queried much more efficiently than the initial risk. On top of providing that generic recipe for learning via surrogate PAC-Bayes bounds, we (i) contribute theoretical results establishing that iteratively optimising our surrogates implies the optimisation of the original generalisation bounds, (ii) instantiate this strategy to the framework of meta-learning, introducing a meta-objective offering a closed form expression for meta-gradient, (iii) illustrate our approach with numerical experiments inspired by an industrial biochemical problem.

PAC-Bayes元学习优化算法泛化分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。