通过粗到精的步骤合并，减少大模型推理中的冗余信息。
Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning
- 先合并相邻步骤形成粗粒度推理块，再逐步细化
- 在两个数据集上提升推理质量，三种损失函数均有效
- 适合需要高质量推理数据的数学推理研究者
过程奖励模型（PRM）在数学推理任务中至关重要，依赖高质量的监督过程数据。然而我们发现，大语言模型生成的推理步骤常缺乏严格的信息递增性，导致冗余，影响有效推理。为此，我们提出一种简单而有效的粗到精策略（CFPRM）：不直接检测冗余步骤，而是先建立粗粒度窗口，将相邻推理步骤合并为统一的整体步骤；随后逐步缩小窗口，提取细粒度推理步骤，实现多粒度数据收集以用于训练。通过这一分层优化过程，CFPRM在缓解冗余的同时保留了关键细粒度知识。在两个推理数据集上，针对三种损失函数的大量实验验证了该方法的有效性与通用性。
原文摘要 · Abstract (English)
The Process Reward Model (PRM) plays a crucial role in mathematical reasoning tasks, requiring high-quality supervised process data. However, we observe that reasoning steps generated by Large Language Models (LLMs) often fail to exhibit strictly incremental information, leading to redundancy that can hinder effective reasoning. To address this issue, we propose CFPRM, a simple yet effective coarse-to-fine strategy. Instead of focusing on the detection of redundant steps, our approach first establishes a coarse-grained window to merge adjacent reasoning steps into unified, holistic steps. The window size is then progressively reduced to extract fine-grained reasoning steps, enabling data collection at multiple granularities for training. By leveraging this hierarchical refinement process, CFPRM mitigates redundancy while preserving essential fine-grained knowledge. Extensive experiments on two reasoning datasets across three loss criteria validate the CFPRM's effectiveness and versatility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。