首个分离评估因果识别与估计的现实世界基准,助力系统精准诊断因果推理缺陷。
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
- 分两步评估:先生成结构化识别方案,再给出点估计与标准误。
- LLM在识别策略上正确率79%,但完整方案正确率仅34%。
- 适合开发更鲁棒的自动因果推断系统的研究者使用。
现有自动化因果推断基准多依赖单一数值输出(如平均处理效应ATE),混淆了因果分析中的两个关键步骤:识别(基于假设制定有效研究设计)与估计(在有限数据上实现该设计)。我们提出CausalReasoningBenchmark,包含173个查询、132个真实世界数据集,源自79篇同行评审论文及三本主流因果推断教材。每个查询要求系统提供:(i) 结构化的识别规范(含策略、处理变量、结果变量、控制变量及设计特异性要素);(ii) 点估计值与标准误。通过分别评分两部分,可实现细粒度诊断,区分因果推理错误与数值执行错误。基于顶尖LLM的基线测试显示,模型虽能正确识别高层次策略79%次,但完整识别规范正确率降至34%,表明瓶颈在于研究设计的细微细节而非计算能力。该基准已公开于Hugging Face,旨在推动更稳健的自动化因果推断系统发展。
原文摘要 · Abstract (English)
Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification - formulating a valid research design under stated assumptions - and estimation - implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 173 queries across 132 real-world datasets, curated from 79 peer-reviewed research papers and three widely-used causal-inference textbooks. For each query a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state of the art LLM show that, while the model correctly identifies the high-level strategy in 79% of cases, full identification-specification correctness drops to only 34%, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。