让AI分析数据时能自我检查错误并区分探索与失误。
Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis
- 设计可主动探测中间状态的环境感知奖励模型,识别隐藏错误。
- 通过三元奖励策略区分可纠正的错误与不可挽回的失误,提升判断准确性。
- 仅用40亿参数就超越强基线,适合需要高可靠性的智能数据分析场景。
过程奖励模型(PRMs)在数学等静态领域显著增强了大语言模型(LLMs)的推理能力,但在动态数据分析任务中潜力尚未充分挖掘。本文首次实证发现,通用领域PRMs难以有效监督数据分析师代理:无法检测无声错误(不触发异常但结果错误),且误罚探索性操作,将必要试错视为基础错误。为此,提出环境感知生成式过程奖励模型DataPRM,具备两项核心能力:(1) 作为主动验证器,自主与环境交互以探测中间执行状态并发现沉默错误;(2) 采用反思感知的三元奖励策略,区分可修正的基础性错误与不可恢复的失误。通过多样性驱动轨迹生成与知识增强的步骤级标注,构建了超8000条高质量训练数据。实验表明,DataPRM在ScienceAgentBench和DABStep上分别使下游策略模型性能提升7.21%和11.28%(Best-of-N推理)。值得注意的是,仅40亿参数的DataPRM即超越强基线,并在多种测试时扩展策略下表现出稳健泛化能力。此外,将其集成至强化学习框架后,在DABench和TableBench上分别达到78.73%和64.84%准确率,显著优于基于结果奖励的基线,验证了过程奖励监督的有效性。代码已开源。
原文摘要 · Abstract (English)
Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs) within static domains such as mathematics. However, their potential in dynamic data analysis tasks remains underexplored. In this work, we first present a empirical study revealing that general-domain PRMs struggle to supervise data analysis agents. Specifically, they fail to detect silent errors, logical flaws that yield incorrect results without triggering interpreter exceptions, and erroneously penalize exploratory actions, mistaking necessary trial-and-error exploration for grounding failures. To bridge this gap, we introduce DataPRM, a novel environment-aware generative process reward model that (1) can serve as an active verifier, autonomously interacting with the environment to probe intermediate execution states and uncover silent errors, and (2) employs a reflection-aware ternary reward strategy that distinguishes between correctable grounding errors and irrecoverable mistakes. We design a scalable pipeline to construct over 8K high-quality training instances for DataPRM via diversity-driven trajectory generation and knowledge-augmented step-level annotation. Experimental results demonstrate that DataPRM improves downstream policy LLMs by 7.21% on ScienceAgentBench and 11.28% on DABStep using Best-of-N inference. Notably, with only 4B parameters, DataPRM outperforms strong baselines, and exhibits robust generalizability across diverse Test-Time Scaling strategies. Furthermore, integrating DataPRM into Reinforcement Learning yields substantial gains over outcome-reward baselines, achieving 78.73% on DABench and 64.84% on TableBench, validating the effectiveness of process reward supervision. Code is available at https://github.com/zjunlp/DataMind.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。