用更稳定的方法让大模型生成更多样化推理路径，性能超越现有技术。
GFlowRL: Scaling Distribution-Matching RL to Large Language Models
- 用批次内蒙特卡洛估计替代复杂分区函数，简化训练流程。
- 在14B模型上达Codeforces 2048分，接近o3-mini表现，多项评测领先。
- 适用于稠密和稀疏架构，最大支持235B参数，首次实现跨架构稳定扩展。
生成流网络（GFlowNets）为大模型推理提供了一种有前景的替代强化学习方法，通过匹配奖励分布而非聚焦单一最优路径，促进多样化推理。尽管在数学与代码任务中已见成效，但将其扩展至现代后训练流水线仍面临挑战：随着模型规模、采样长度、奖励噪声及分布式系统复杂度同步上升，原本作为归一化器的可学习提示条件分区函数反而成为梯度不稳与工程负担的源头。通过系统分析，我们发现该分区函数可被移除，改由训练所需采样组内已有的批次蒙特卡洛估计替代。为此提出GFlowRL，一种精简版的GFlowNet风格强化学习算法，完全舍弃辅助分区网络，同时保留奖励分布匹配目标，并引入两项稳定机制：重要性采样校正以应对采样/训练漂移，以及非对称流差截断处理异常残差。GFlowRL在数学、代码及对抗红队测试中全面超越现有方法，14B模型达到Codeforces 2048分（距o3-mini仅差25 Elo），在AdvBench与HarmBench上取得最高ASR@1平均值，优于先前最先进多轮攻击模型；而在流动强化学习（FlowRL）出现发散的场景下，其仍能收敛。该方案成功迁移至全部评估的MoE配置，最大达235B参数，而原版FlowRL在此规模下无法收敛。据我们所知，GFlowRL是首个在密集与稀疏架构间均实现稳定扩展的GFlowNet风格强化学习算法。代码将公开于：https://github.com/microsoft/gflowrl
原文摘要 · Abstract (English)
Generative Flow Networks (GFlowNets) offer a promising alternative to reward-maximizing reinforcement learning (RL) for large reasoning models, encouraging diverse reasoning paths by matching reward distributions rather than collapsing to dominant modes. Recent work shows promise on math and code, but scaling GFlowNet-style RL to modern post-training pipelines remains difficult: as model size, rollout horizon, reward noise, and distributed-systems complexity grow together, a learned prompt-conditional partition function becomes a source of gradient instability and engineering overhead rather than a useful normalizer. Through systematic analysis, we find that the learned partition function, previously treated as essential, can be replaced by an in-batch Monte Carlo estimate computed from the rollout group already required for training. We propose GFlowRL, a streamlined GFlowNet-style RL algorithm that removes the auxiliary partition network entirely while preserving the reward-distribution-matching objective, completed by two stabilizers: importance-sampling correction for rollout/trainer drift and asymmetric flow-gap clipping for outlier residuals. GFlowRL exceeds all counterparts on math, code, and adversarial red-teaming benchmarks, reaching a Codeforces rating of 2048 at the 14B scale (within 25 Elo of o3-mini) and attaining the highest average ASR@1 on AdvBench and HarmBench, outperforming the previous SOTA multi-turn attacker in a regime where FlowRL, a prior GFlowNet-style method, diverges. The same recipe transfers to all evaluated MoE configurations up to 235B parameters, where FlowRL again fails to converge. To our knowledge, GFlowRL is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures. Code will be at: https://github.com/microsoft/gflowrl
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。