无需代理模型,通过轨迹提炼实现高效离线生成流网络训练
Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Training
- 用逆强化学习从离线轨迹中提取细粒度奖励,指导探索
- 在多个数据集上收敛更快,采样质量显著优于基线方法
- 适合数据稀缺场景下的生成模型训练,尤其适用于高成本评估任务
生成流网络(GFlowNets)在采样多样且高回报对象方面表现优异。但在无法进行主动奖励查询的实际应用中,这类模型需依赖静态离线数据集进行训练。现有方法通常依赖代理模型提供在线采样轨迹的奖励反馈,但受制于数据稀疏或评估成本高,构建可靠代理困难。尽管已有无代理方法尝试解决该问题,但常施加粗略约束,限制了模型的有效探索能力。为此,本文提出轨迹提炼生成流网络(TD-GFN),一种新型无代理训练框架。TD-GFN利用逆强化学习(IRL)从离线轨迹中提取细粒度、逐转移的边奖励,为高效探索提供丰富结构引导。关键在于,通过图结构剪枝与优先反向采样间接引导策略,确保梯度更新仅依赖数据集中真实终端奖励,从而避免误差传播。实验证明,TD-GFN在收敛速度和样本质量上均显著优于多种基线方法,建立了一种更鲁棒高效的离线GFlowNet训练范式。
原文摘要 · Abstract (English)
Generative Flow Networks (GFlowNets) excel at sampling diverse, high-reward objects. In many practical applications where active reward queries are infeasible, these models must be trained using static offline datasets. Prevailing training methods typically rely on a proxy model to provide reward feedback for online sampled trajectories. However, constructing a reliable proxy is often challenging due to data scarcity or high evaluation costs. While existing proxy-free approaches attempt to address this, they often impose coarse constraints that limit the model's ability to explore effectively. To overcome these limitations, we propose Trajectory-Distilled GFlowNet (TD-GFN), a novel proxy-free training framework. TD-GFN utilizes inverse reinforcement learning (IRL) to extract dense, transition-level edge rewards from offline trajectories, providing rich structural guidance for efficient exploration. Crucially, to ensure robustness, these rewards guide the policy indirectly through DAG pruning and prioritized backward sampling. This design ensures that gradient updates rely exclusively on ground-truth terminal rewards from the dataset, thereby preventing error propagation. Empirical results demonstrate that TD-GFN significantly outperforms a broad range of existing baselines in both convergence speed and sample quality, establishing a more robust and efficient paradigm for offline GFlowNet training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。