让大模型推理过程更可信：新奖励模型可精准评估长链思维轨迹。
ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
- 设计轨迹感知的奖励机制，同时监督每一步和整体推理路径。
- 在多项难题上提升12.1%（微调）、4.5%（强化学习）和6.3%（测试时扩展）。
- 适用于小模型训练、强化学习优化及边缘部署，支持多种场景。
过程奖励模型（PRMs）已成为监督大语言模型中间推理步骤的有效框架。现有PRMs主要基于模型最终输出训练，难以稳健评估前沿推理模型（如Deepseek-R1）生成的轨迹-响应型推理痕迹。本文提出ReasonFlux-PRM，一种专为轨迹-响应类推理迹线设计的新型轨迹感知奖励模型。该模型结合步骤级与轨迹级监督，实现与结构化思维链数据对齐的细粒度奖励分配。我们将其适配于离线与在线两种设置：(i) 用于筛选高质量模型蒸馏数据以微调小型模型；(ii) 为强化学习中的策略优化提供密集的过程级奖励；(iii) 支持奖励引导的Best-of-N测试时扩展。在AIME、MATH500和GPQA-Diamond等挑战性基准上的实验表明，ReasonFlux-PRM-7B比强基线模型（如Qwen2.5-Math-PRM-72B）和人工标注基准选择更优质数据。由此得到的ReasonFlux-PRM-7B在不同任务中持续提升性能，平均提升分别为12.1%（微调）、4.5%（强化学习）和6.3%（测试时扩展）。我们还发布了适用于资源受限场景与边缘部署的高效版本ReasonFlux-PRM-1.5B。
原文摘要 · Abstract (English)
Process Reward Models (PRMs) have recently emerged as a powerful framework for supervising intermediate reasoning steps in large language models (LLMs). Previous PRMs are primarily trained on model final output responses and struggle to evaluate intermediate thinking trajectories robustly, especially in the emerging setting of trajectory-response outputs generated by frontier reasoning models like Deepseek-R1. In this work, we introduce ReasonFlux-PRM, a novel trajectory-aware PRM explicitly designed to evaluate the trajectory-response type of reasoning traces. ReasonFlux-PRM incorporates both step-level and trajectory-level supervision, enabling fine-grained reward assignment aligned with structured chain-of-thought data. We adapt ReasonFlux-PRM to support reward supervision under both offline and online settings, including (i) selecting high-quality model distillation data for downstream supervised fine-tuning of smaller models, (ii) providing dense process-level rewards for policy optimization during reinforcement learning, and (iii) enabling reward-guided Best-of-N test-time scaling. Empirical results on challenging downstream benchmarks such as AIME, MATH500, and GPQA-Diamond demonstrate that ReasonFlux-PRM-7B selects higher quality data than strong PRMs (e.g., Qwen2.5-Math-PRM-72B) and human-curated baselines. Furthermore, our derived ReasonFlux-PRM-7B yields consistent performance improvements, achieving average gains of 12.1% in supervised fine-tuning, 4.5% in reinforcement learning, and 6.3% in test-time scaling. We also release our efficient ReasonFlux-PRM-1.5B for resource-constrained applications and edge deployment. Project: https://github.com/Gen-Verse/ReasonFlux
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。