用自适应教师指导学生采样,提升探索效率与模式覆盖
Adaptive teachers for amortized samplers
- 引入自适应教师模型,动态聚焦学生未探索区域
- 在扩散模型和生化任务中提升采样效率与模式覆盖率
- 适合需要高效探索的生成采样场景,如分子设计
摊销推断是训练参数化模型(如神经网络)以近似具有给定非归一化密度的分布,当精确采样不可行时。当采样被建模为序列决策过程时,可使用强化学习方法(如生成流网络)训练采样策略。离策略强化学习训练有助于发现多样且高回报的候选样本,但现有方法在高效探索方面仍存在挑战。本文提出使用自适应训练分布(即“教师”)来引导主摊销采样器(即“学生”)的训练。该教师是一个辅助行为模型,被训练用于采样学生模型的高损失区域,并能泛化至未探索的模式,从而通过提供高效的训练课程提升模式覆盖率。我们在一个设计用于呈现探索挑战的合成环境、两个基于扩散的采样任务以及四个生化发现任务中验证了该方法的有效性,结果表明其显著提升了样本效率和模式覆盖率。源代码见 https://github.com/alstn12088/adaptive-teacher。
原文摘要 · Abstract (English)
Amortized inference is the task of training a parametric model, such as a neural network, to approximate a distribution with a given unnormalized density where exact sampling is intractable. When sampling is implemented as a sequential decision-making process, reinforcement learning (RL) methods, such as generative flow networks, can be used to train the sampling policy. Off-policy RL training facilitates the discovery of diverse, high-reward candidates, but existing methods still face challenges in efficient exploration. We propose to use an adaptive training distribution (the \teacher) to guide the training of the primary amortized sampler (the \student). The \teacher, an auxiliary behavior model, is trained to sample high-loss regions of the \student and can generalize across unexplored modes, thereby enhancing mode coverage by providing an efficient training curriculum. We validate the effectiveness of this approach in a synthetic environment designed to present an exploration challenge, two diffusion-based sampling tasks, and four biochemical discovery tasks demonstrating its ability to improve sample efficiency and mode coverage. Source code is available at https://github.com/alstn12088/adaptive-teacher.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。