arXiv:2501.18528cs.LGstat.ML2025-01ICML被引 3

联合学习能量模型与分区函数,实现高效无采样的概率建模。

Joint Learning of Energy-based Models and their Partition Function

  • 用神经网络同时学习能量函数和对数分区函数
  • 无需MCMC即可通过梯度下降训练,且能估计未见数据的分区函数
  • 适用于组合大空间的标签排序等任务,理论可逼近最优最大似然

能量模型(EBM)为使用神经网络参数化概率分布提供了灵活框架。然而,由于需要计算分区函数(归一化常数),通过精确最大似然估计(MLE)学习EBM通常不可行。本文提出一种新方法,用于在组合性大的离散空间(如集合或排列)中近似学习概率型EBM。核心思想是联合学习一个能量模型及其对数分区函数,两者均以神经网络参数化。该方法不仅提供了一种可计算的目标函数,支持通过随机梯度下降训练(无需依赖马尔可夫链蒙特卡洛,MCMC),还提供了一种在未见数据点上估计对数分区函数的新方式。理论上,我们证明当在连续函数空间中优化时,该方法可恢复最优的MLE解。此外,该方法自然推广至Fenchel-Young损失族,首次实现了在组合大空间中可计算地优化sparsemax损失。我们在多标签分类和标签排序任务上验证了该方法的有效性。

原文摘要 · Abstract (English)

Energy-based models (EBMs) offer a flexible framework for parameterizing probability distributions using neural networks. However, learning EBMs by exact maximum likelihood estimation (MLE) is generally intractable, due to the need to compute the partition function (normalization constant). In this paper, we propose a novel formulation for approximately learning probabilistic EBMs in combinatorially-large discrete spaces, such as sets or permutations. Our key idea is to jointly learn both an energy model and its log-partition, both parameterized as a neural network. Our approach not only provides a novel tractable objective criterion to learn EBMs by stochastic gradient descent (without relying on MCMC), but also a novel means to estimate the log-partition function on unseen data points. On the theoretical side, we show that our approach recovers the optimal MLE solution when optimizing in the space of continuous functions. Furthermore, we show that our approach naturally extends to the broader family of Fenchel-Young losses, allowing us to obtain the first tractable method for optimizing the sparsemax loss in combinatorially-large spaces. We demonstrate our approach on multilabel classification and label ranking.

能量模型概率建模神经网络分区函数

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。