arXiv:2410.02596cs.LGcs.AI2024-10ICLR被引 8

提出三种新损失函数,提升生成流网络的训练效率与多样性。

Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks

  • 基于发散度理论设计新型回归损失,区分探索与利用行为。
  • 在三个基准任务中显著加速收敛,提升样本多样性和鲁棒性。
  • 适用于主流训练算法,适合需要高质量生成的场景。

生成流网络(GFlowNets)是一类新型生成模型,用于从非归一化分布中采样,在多个重要任务中表现出色,其训练算法受到广泛关注。现有方法主要关注训练对象选择、参数化、采样策略和反向策略,以优化信用分配、探索或利用。然而,回归损失的选择——直接影响训练策略的探索与利用行为——长期被忽视。由于缺乏理论指导,多数算法采用对数空间中的平方误差损失。本文严格证明:不同回归损失对应特定发散度度量,可据此设计并分析损失函数。重点考察零强制(促进高奖励利用)与零避免(鼓励探索、增强多样性)两种性质。基于此,提出三种新损失:Shifted-Cosh、Linex(1/2) 和 Linex(1)。在超网格、位序列生成和分子生成三个基准上评估,所提损失兼容多数现有训练算法,显著提升收敛速度、样本多样性和鲁棒性。

原文摘要 · Abstract (English)

Generative Flow Networks (GFlowNets) are a novel class of generative models designed to sample from unnormalized distributions and have found applications in various important tasks, attracting great research interest in their training algorithms. In general, GFlowNets are trained by fitting the forward flow to the backward flow on sampled training objects. Prior work focused on the choice of training objects, parameterizations, sampling and resampling strategies, and backward policies, aiming to enhance credit assignment, exploration, or exploitation of the training process. However, the choice of regression loss, which can highly influence the exploration and exploitation behavior of the under-training policy, has been overlooked. Due to the lack of theoretical understanding for choosing an appropriate regression loss, most existing algorithms train the flow network by minimizing the squared error of the forward and backward flows in log-space, i.e., using the quadratic regression loss. In this work, we rigorously prove that distinct regression losses correspond to specific divergence measures, enabling us to design and analyze regression losses according to the desired properties of the corresponding divergence measures. Specifically, we examine two key properties: zero-forcing and zero-avoiding, where the former promotes exploitation and higher rewards, and the latter encourages exploration and enhances diversity. Based on our theoretical framework, we propose three novel regression losses, namely, Shifted-Cosh, Linex(1/2), and Linex(1). We evaluate them across three benchmarks: hyper-grid, bit-sequence generation, and molecule generation. Our proposed losses are compatible with most existing training algorithms, and significantly improve the performances of the algorithms concerning convergence speed, sample diversity, and robustness.

生成模型损失函数流网络训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。