通过替换前提谓词，检测大模型推理是否真正依赖前提。
Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution
- 黑盒测试方法：逐步替换前提谓词，观察推理结论是否变化。
- 在ProntoQA上检测依赖关系，F1达0.806，远超基线。
- 发现66%正确解答存在错误推理，适合验证模型逻辑可靠性。
大型语言模型生成的链式思维（CoT）看似逻辑严密，但可能并不真正依赖其陈述的前提。本文提出干预性基础审计（Interventional Grounding Audits），一种黑盒、步骤级的前提依赖性测试：将单个前提的靶向谓词替换为新符号，重新运行模型，并检查每个推理步骤的归一化结论（标准谓词形式）是否改变。我们在ProntoQA上进行评估，这是一个具有真实证明树的合成多跳演绎推理基准，已知每一步的前提依赖关系。对50个ProntoQA问题使用GPT-4o，该方法在检测证明树依赖关系上的F1为0.806（谓词决定型依赖的F1为0.885，召回率为100%），显著优于自一致性基线（F1=0.343；95%置信区间无重叠）。我们进一步发现，在所有正确求解的问题中，有66%包含至少一个在一致替换下对直接证明树依赖不敏感的对齐步骤——这些均涉及实体引入前提，是此前一致替换评估器的已知盲点，揭示了‘答案正确但推理错误’的信号，被动方法无法察觉。所有审计证书、原始输出和复现脚本均已公开于公共GitHub仓库，并讨论了在形式化可解析基准之外的应用局限。
原文摘要 · Abstract (English)
Large language models produce chain-of-thought (CoT) reasoning that appears logically sound yet may not genuinely depend on its stated premises. We introduce interventional grounding audits, a black-box, step-level test of premise dependency: we intervene on a single premise by substituting its target predicate with a fresh symbol, re-run the model, and check whether each reasoning step's normalized conclusion (canonical predicate form) changes. We evaluate on ProntoQA, a synthetic multi-hop deductive reasoning benchmark with gold proof trees, where step-level premise dependencies are known. Applied to 50 ProntoQA problems with GPT-4o, our method achieves F1 = 0.806 on detecting proof-tree dependencies (F1 = 0.885 on predicate-determining dependencies; Recall = 100%), significantly outperforming a self-consistency baseline (F1 = 0.343; 95% bootstrap CIs non-overlapping). We further identify that 66% of correctly-solved problems contain at least one aligned step insensitive to a direct proof-tree dependency under consistent substitution -- all involving entity-introduction premises, a documented blind spot of the consistent-substitution evaluator -- a "right answer, wrong reasoning" signal invisible to passive methods. All audit certificates, raw outputs, and reproduction scripts are available in a public GitHub repository, and we discuss scope limits beyond formal, parsable benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。