arXiv:2510.03185cs.LG2025-10被引 3

用因果图评估物理推理过程,更准地发现模型错误。

PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning

  • 用有向无环图表示公式间的因果关系,实现细粒度评分。
  • 相比人类专家评分,新方法相关性提升23.7%。
  • 适合研究物理推理模型缺陷或改进评估标准的人。

现有竞赛式物理推理评测仅关注最终答案,无法捕捉推理过程;近期的逐步评分方法依赖启发式大模型打分或线性假设,可靠性不足。我们提出PRISM-Physics,一个基于因果有向无环图(DAG)的物理推理过程级评估框架与基准。解题过程以公式间的因果依赖关系表示为DAG,实现可解释、理论严谨的评分。我们证明了该表示及其评分策略的最优性,并结合自研的完全基于规则的符号等价匹配方法,确保对多种表达形式的一致验证,无需启发式判断。实验表明,该框架与人类专家评分更具一致性(相关性达0.89)。在主流大模型上的测试揭示了其在物理推理中持续存在的根本性错误,而逐步评分提供了诊断性洞察与丰富训练信号。通过结构严谨性、理论保证与符号验证的结合,PRISM-Physics为推进过程级评估和培养具备深层科学推理能力的模型奠定了原则性基础。

原文摘要 · Abstract (English)

Benchmarks for competition-style reasoning have advanced evaluation in mathematics and programming, yet physics remains comparatively explored. Most existing physics benchmarks evaluate only final answers, which fail to capture reasoning processes, while recent stepwise methods rely on heuristic LLM-as-judge scoring or restrictive linear assumptions, limiting reliability and diagnostic validity. We introduce PRISM-Physics, a process-level evaluation framework and benchmark for complex physics reasoning problems. Solutions are represented as directed acyclic graphs (DAGs) of formulas, explicitly encoding causal dependencies among intermediate steps to enable fine-grained, interpretable, and theoretically grounded scoring. We prove the optimality of the DAG representation and the corresponding scoring policy. Combining with a fully rule-based method for symbolic formula equivalence matching that we developed, we ensure consistent validation across diverse formulations without heuristic judgments. Results show that our evaluation framework is more aligned with human experts' scoring. Experiments on state-of-the-art LLMs reveal persistent reasoning failures in physics, while step-level scoring offers both diagnostic insight and rich signals for later training. By combining structural rigor, theoretical guarantees, and symbolic validation, PRISM-Physics provides a principled foundation for advancing process-level evaluation and guiding the development of models with deeper scientific reasoning capabilities.

物理推理因果图评估框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。