用连续流建模价值，让大模型训练更稳定泛化。
DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training
- 将价值建模为时间上的连续流，捕捉更精细的状态信息
- 在噪声监督下训练更稳定，对话/数学/科学任务均优于基线
- 适合需要鲁棒性与泛化能力的复杂任务后训练场景
在真实环境中训练强化学习系统仍面临监督信号噪声大、域外泛化差的问题，尤其在大语言模型后训练中更为显著。现有分布式强化学习方法通过多分位点建模价值提升鲁棒性，但各分位点独立学习为标量，导致价值表示粗糙，难以适应复杂和域外条件。本文提出DFPO（基于条件风险与一致性控制的分布值流策略优化），将价值建模为跨时间步的连续流场，通过学习价值流场实现更精细的状态条件化，从而提升优势估计精度。为应对噪声反馈，进一步引入沿价值流轨迹的条件风险控制与一致性约束，增强训练稳定性。在对话、数学推理和科学任务上的实验表明，DFPO在噪声监督下优于PPO、FlowRL等鲁棒基线，在训练稳定性和泛化能力上均有提升。
原文摘要 · Abstract (English)
Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain (OOD) generalization, especially in LLM post-training. Recent distributional RL methods improve robustness by modeling values with multiple quantile points, but they still learn each quantile independently as a scalar. This results in rough-grained value representations that lack fine-grained conditioning on state information, struggling under complex and OOD conditions. We propose DFPO (Distributional Value Flow Policy Optimization with Conditional Risk and Consistency Control), a robust distributional RL framework that models values as continuous flows across time steps. By scaling value modeling through learning of a value flow field instead of isolated quantile predictions, DFPO captures richer state information for more accurate advantage estimation. To stabilize training under noisy feedback, DFPO further integrates conditional risk control and consistency constraints along value flow trajectories. Experiments on dialogue, math reasoning, and scientific tasks show that DFPO outperforms PPO, FlowRL, and other robust baselines under noisy supervision, achieving improved training stability and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。