通过匹配奖励分布提升大模型推理多样性，避免过度优化单一路径。
FlowRL: Matching Reward Distributions for LLM Reasoning
- 用可学习的分区间函数将奖励转为分布，以反KL散度优化策略。
- 数学任务上比GRPO提升10.0%，比PPO提升5.1%，代码任务也更优。
- 适合追求多样化、鲁棒性推理的LLM强化学习研究者。
我们提出FlowRL：在大语言模型强化学习中，通过流平衡匹配完整的奖励分布，而非仅最大化奖励。当前先进推理模型多采用奖励最大化方法（如PPO和GRPO），容易过度优化主导奖励信号，忽视较少出现但有效的推理路径，导致多样性下降。相比之下，我们使用可学习的分区函数将标量奖励转化为归一化目标分布，并最小化策略与目标分布之间的反KL散度。我们实现该思想为一种流平衡优化方法，促进多样化探索和可泛化推理轨迹。我们在数学和代码推理任务上进行实验：FlowRL在数学基准上相对于GRPO平均提升10.0%，相对于PPO提升5.1%，在代码推理任务上表现也持续更优。结果表明，奖励分布匹配是实现高效探索和多样化推理的关键步骤。
原文摘要 · Abstract (English)
We propose FlowRL: matching the full reward distribution via flow balancing instead of maximizing rewards in large language model (LLM) reinforcement learning (RL). Recent advanced reasoning models adopt reward-maximizing methods (\eg, PPO and GRPO), which tend to over-optimize dominant reward signals while neglecting less frequent but valid reasoning paths, thus reducing diversity. In contrast, we transform scalar rewards into a normalized target distribution using a learnable partition function, and then minimize the reverse KL divergence between the policy and the target distribution. We implement this idea as a flow-balanced optimization method that promotes diverse exploration and generalizable reasoning trajectories. We conduct experiments on math and code reasoning tasks: FlowRL achieves a significant average improvement of $10.0\%$ over GRPO and $5.1\%$ over PPO on math benchmarks, and performs consistently better on code reasoning tasks. These results highlight reward distribution-matching as a key step toward efficient exploration and diverse reasoning in LLM reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。