测试大模型是否真懂因果，发现去掉关键词后准确率暴跌
Caliper: Probing Lexical Anchors versus Causal Structure in LLMs
- 用占位符替换变量名，保留因果结构但移除词汇线索
- 准确率平均下降17.6个百分点，部分模型降幅超29个百分点
- 多数模型依赖关键词而非逻辑推理，适合评估模型本质能力
大语言模型在因果推理基准测试中准确率达50%至70%，但其表现是基于结构化推理还是词汇模式匹配尚不明确。本文提出Caliper，一种受控扰动方法，将语义变量名替换为占位符，同时保留问题的因果图与概率设定。在九个指令微调的LLM（3.8B至671B）和三个因果推理基准上，词汇匿名化导致准确率显著下降：在3.8B-14B小模型集上分别下降+7.6、+27.0、+11.1个百分点；在CRASS和e-CARE上，九个前沿模型（2024–2026年生成）分别下降+29.6和+18.0个百分点。40个模型-基准组合中有39个出现正向差距，且在CLadder伪词子集上差距缩小17倍。结构支架和少样本上下文学习虽能缩小差距，但主要通过降低小模型的基准准确率实现，而非恢复性能。零样本评估下，当前指令微调的LLM在移除词汇锚点后几乎无结构性因果推理证据。
原文摘要 · Abstract (English)
Large language models reach 50 to 70% accuracy on causal reasoning benchmarks such as CLadder, but it is unclear whether this reflects structural reasoning or lexical pattern matching. We introduce Caliper, a controlled perturbation that replaces semantic variable names with placeholder tokens while preserving the causal graph and probabilistic specification of each question. Across nine instruction-tuned LLMs from 3.8B to 671B and three causal reasoning benchmarks, lexical anonymization yields robust accuracy drops of +7.6, +27.0, and +11.1 pp on a local 3.8B-14B set, rising to +29.6 and +18.0 pp on CRASS and e-CARE across nine frontier models spanning the 2024-2026 generations. Of 40 engaged model-by-benchmark cells, 39 show a positive gap, and the gap collapses by 17x on CLadder's pseudoword subset. Structured scaffolding and few-shot in-context learning each narrow the gap, but mainly by lowering P0 accuracy on smaller models rather than recovering P1. Current instruction-tuned LLMs, evaluated zero-shot, show little evidence of structural causal reasoning once lexical anchors are removed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。