提出更高效的训练方法,让生成流网络收敛更快更稳定。
On Divergence Measures for Training GFlowNets
- 用四种新发散度替代传统损失,提升训练稳定性
- 设计控制变量使梯度方差降低,加速模型收敛
- 适合研究生成模型优化与变分推断的学者
生成流网络(GFlowNets)是用于从可组合对象的非归一化分布中采样的近似推断模型,广泛应用于因果发现、自然语言处理和药物发现等领域。传统训练方法通过最小化前向策略与后向策略之间期望对数平方差来实现流匹配,但直接使用标准KL散度会导致偏差大、方差高的估计问题。本文系统回顾了四种发散度:Rényi-α、Tsallis-α、反向KL和正向KL,并为它们在训练GFlowNets时设计出统计效率更高的随机梯度估计器。验证表明,正确最小化这些发散度可获得理论正确且实证有效的训练方案,通常显著加快收敛速度。为此,我们基于REINFORCE留一法与得分匹配估计器设计了控制变量,有效降低学习目标梯度的方差。本工作缩小了GFlowNets训练与广义变分近似之间的差距,为基于发散度最小化的算法设计提供新思路。
原文摘要 · Abstract (English)
Generative Flow Networks (GFlowNets) are amortized inference models designed to sample from unnormalized distributions over composable objects, with applications in generative modeling for tasks in fields such as causal discovery, NLP, and drug discovery. Traditionally, the training procedure for GFlowNets seeks to minimize the expected log-squared difference between a proposal (forward policy) and a target (backward policy) distribution, which enforces certain flow-matching conditions. While this training procedure is closely related to variational inference (VI), directly attempting standard Kullback-Leibler (KL) divergence minimization can lead to proven biased and potentially high-variance estimators. Therefore, we first review four divergence measures, namely, Renyi-$α$'s, Tsallis-$α$'s, reverse and forward KL's, and design statistically efficient estimators for their stochastic gradients in the context of training GFlowNets. Then, we verify that properly minimizing these divergences yields a provably correct and empirically effective training scheme, often leading to significantly faster convergence than previously proposed optimization. To achieve this, we design control variates based on the REINFORCE leave-one-out and score-matching estimators to reduce the variance of the learning objectives' gradients. Our work contributes by narrowing the gap between GFlowNets training and generalized variational approximations, paving the way for algorithmic ideas informed by the divergence minimization viewpoint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。