首个基于实际执行的因果推断评估基准，验证大模型能否正确实现经济研究中的因果分析流程。
CausalVerify: An Execution-Grounded Benchmark for LLM Causal Inference Workflows
- 构建真实论文与合成数据配对的执行验证框架，区分可验证计算与主观解释。
- 7个大模型在标准容差下仅10%至88%通过执行验证，15.5%的可运行流程得出错误结论。
- 执行结果比文本描述更可靠，但模型自信度无法有效区分对错，适合评估特定因果方法能力。
现有大模型因果推断评测多依赖方法描述或代码是否运行，而非实际工作流能否恢复目标因果估计。CausalVerify通过分离真实解释与可验证计算，针对结构化计量经济学因果估计流程提出执行验证问题。该基准将259篇已发表经济学论文（重构研究问题、数据描述、制度背景）与100个固定种子的合成场景配对，生成用于双重差分、事件研究、工具变量和断点回归设计的CSV数据集。实验A（真实论文文本一致性）评估四类大模型共识下方法族与方向的一致性；实验B（合成执行）运行模型生成的R代码，检查提取的处理效应估计是否与同一数据集上的经典估计器一致，此执行验证层为L2b+，区别于仅记录代码能否运行的L2b。校准组测试自报置信度是否能区分正确与错误流程。实验B中，七种大模型的L2b+通过率在10%至88%之间（默认50%容差），426个可执行工作流中有66个（15.5%）给出错误估计。执行排名（L2b）与L2b+高度相关（Kendall τ=0.81，Spearman ρ=0.93），远优于文本方向评分（L4）（Kendall τ介于-0.20至0.10）。Llama-3.3-70B-Instruct同样显示相同质性差距，且自报置信度无法可靠区分对错。结论限于四种设计类型下的标准化单次工作流，评估范围不涵盖一般因果推断能力。代码、数据、缓存输出及数据表均已发布。
原文摘要 · Abstract (English)
Existing causal-inference benchmarks for LLMs mostly score method descriptions or whether generated code runs, not whether the executed workflow recovers the target causal estimate. CausalVerify studies this verification problem for structured econometric causal-estimation workflows by separating realistic interpretation from verifiable computation. It pairs 259 published economics papers (reconstructed research question, data description, institutional context) with 100 fixed-seed synthetic scenarios that realise CSV datasets for difference-in-differences, event study, instrumental variables, and regression discontinuity designs. Experiment A (real-paper text agreement) scores method-family and direction agreement against four-LLM consensus labels. Experiment B (synthetic execution) runs model-written R code and checks whether the extracted treatment-effect estimate matches a canonical estimator on the same realised dataset; this execution-grounded correctness layer is L2b+, distinct from L2b, which records only whether the code executes. A calibration arm asks whether self-reported confidence separates correct from incorrect workflows. On Experiment B, seven LLMs reach L2b+ pass rates of 10% to 88% at the default 50% tolerance, and 66 of the 426 workflows that execute (15.5%) return a wrong estimate. Execution ranking (L2b) agrees with L2b+ far better than text-direction scoring (L4): Kendall $\tau=0.81$ and Spearman $\rho=0.93$, versus Kendall $\tau$ between $-0.20$ and $0.10$ for L4. Llama-3.3-70B-Instruct shows the same qualitative gap, and reported confidence does not reliably separate correct from incorrect workflows. The claims are confined to standardized single-shot workflows in these four design families under the evaluated R backend and model panel; the benchmark does not measure general causal-inference ability. Code, data, cached outputs, and a datasheet are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。