首个统一框架评估大模型多层级因果推理能力
METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models
- 构建统一上下文下的三层因果阶梯评测体系
- 发现模型在高层级因果推理中性能显著下降
- 揭示干扰信息与上下文忠实度下降两大失效原因
上下文因果推理是大型语言模型的关键但极具挑战的能力。现有基准多在碎片化场景下评估，难以保证上下文一致性或覆盖完整的因果层次。为此，我们提出METER，首次在统一上下文设置下系统性地对大模型在因果阶梯三个层级上的表现进行评测。对多种大模型的广泛评估显示，随着任务在因果层级中的上升，模型能力显著下降。为诊断该退化现象，我们通过错误模式识别和内部信息流追踪开展深入机制分析，发现两类主要失败模式：(1) 模型在低层级因果任务中易受无关但事实正确的信息干扰；(2) 随着任务层级上升，模型对给定上下文的忠实度降低，导致性能下降。我们认为本工作深化了对大模型上下文因果推理机制的理解，并为后续研究奠定了关键基础。代码与数据集已公开于https://github.com/SCUNLP/METER。
原文摘要 · Abstract (English)
Contextual causal reasoning is a critical yet challenging capability for Large Language Models (LLMs). Existing benchmarks, however, often evaluate this skill in fragmented settings, failing to ensure context consistency or cover the full causal hierarchy. To address this, we pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder under a unified context setting. Our extensive evaluation of various LLMs reveals a significant decline in proficiency as tasks ascend the causal hierarchy. To diagnose this degradation, we conduct a deep mechanistic analysis via both error pattern identification and internal information flow tracing. Our analysis reveals two primary failure modes: (1) LLMs are susceptible to distraction by causally irrelevant but factually correct information at lower level of causality; and (2) as tasks ascend the causal hierarchy, faithfulness to the provided context degrades, leading to a reduced performance. We belive our work advances our understanding of the mechanisms behind LLM contextual causal reasoning and establishes a critical foundation for future research. Our code and dataset are available at https://github.com/SCUNLP/METER .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。