测试大模型从真实文本中推断因果关系的能力,发现表现有限。
Can Large Language Models Infer Causal Relationships from Real-World Text?
- 构建首个基于真实学术文献的因果推理基准数据集
- 最佳模型平均F₁仅0.535,显示大模型在此任务上仍困难
- 适用于研究大模型因果推理能力的学者与开发者
理解并从文本中推断因果关系是人类认知的核心,也是推动大语言模型向通用人工智能发展的关键。现有评估工作主要依赖于合成或简化文本,其中因果关系明确且数量少,无法反映真实世界推理的复杂性。本文研究大模型是否能从真实文本中推断因果关系,构建了一个源自真实学术文献的基准数据集,涵盖不同长度、复杂度(显性程度、因果事件与关系数量)和领域的文本。据我们所知,这是首个针对该任务的真实世界数据集。实验表明,大模型在真实文本中推断因果关系面临显著挑战,最佳模型平均F₁仅为0.535。通过分析文本显性、因果事件与关系数量、文本长度及领域等维度,该基准为提升大模型因果推理能力提供了针对性洞见。代码与数据集详见 https://github.com/Ryan-Saklad/ReCITE。
原文摘要 · Abstract (English)
Understanding and inferring causal relationships from texts is a core aspect of human cognition and is essential for advancing large language models (LLMs) towards artificial general intelligence. Existing work evaluating LLM causal reasoning primarily relies on synthetic or simplified texts with explicitly stated causal relationships. These texts typically feature short passages and few causal relations, failing to reflect the complexities of real-world reasoning. In this paper, we investigate whether LLMs are capable of inferring causal relationships from real-world texts. We develop a benchmark drawn from real-world academic literature, which includes diverse texts with respect to length, complexity (different levels of explicitness, number of causal events and relationships), and domain. To the best of our knowledge, our benchmark is the first-ever real-world dataset for this task. Our experiments on this dataset show that LLMs face significant challenges in inferring causal relationships from real-world text, with the best-performing model achieving an average F$_1$ score of only 0.535. Through systematic analysis across aspects of real-world text (explicitness, number of causal events and relationships, length of text, domain), our benchmark offers targeted insights for further research into advancing LLM causal reasoning. Our code and dataset can be found at https://github.com/Ryan-Saklad/ReCITE .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。