构建抽象因果事件基准,提升模型对日常事件因果的理解能力。
ACCESS : A Benchmark for Abstract Causal Event Discovery and Reasoning
- 从大尺度常识数据中提取抽象事件因果对,构建新基准
- 1,400组因果对验证模型在抽象层面的因果推理能力
- 适用于提升大模型问答中的因果推理性能
识别因果关系对于理解现实世界动态和实现因果推理至关重要。现有NLP方法在识别事件因果关系时,因基准规模有限且过度依赖词汇线索,在分布外场景下表现不佳。现代基准受概率因果推断启发,尝试以因果图形式构建事件因果知识表示,其中 exttt{CRAB}是代表性工作。本文提出 exttt{ACCESS},一个面向抽象因果事件发现与推理的基准。不同于以往资源, exttt{ACCESS}聚焦日常事件在抽象层面的因果关系。我们基于 exttt{GLUCOSE}(一个大规模隐含常识因果知识数据集)设计了一套抽象化事件泛化识别流程,从中提取出1,400组因果对。实验表明,当前统计方法和/或大语言模型在自动抽象识别与因果发现上仍面临挑战。然而,我们证明 exttt{ACCESS}提供的抽象因果知识可有效提升大模型在问答任务中的推理表现。
原文摘要 · Abstract (English)
Identifying cause-and-effect relationships is critical to understanding real-world dynamics and ultimately causal reasoning. Existing methods for identifying event causality in NLP, including those based on Large Language Models (LLMs), exhibit difficulties in out-of-distribution settings due to the limited scale and heavy reliance on lexical cues within available benchmarks. Modern benchmarks, inspired by probabilistic causal inference, have attempted to construct causal graphs of events as a robust representation of causal knowledge, where \texttt{CRAB} \citep{romanou2023crab} is one such recent benchmark along this line. In this paper, we introduce \texttt{ACCESS}, a benchmark designed for discovery and reasoning over abstract causal events. Unlike existing resources, \texttt{ACCESS} focuses on causality of everyday life events on the abstraction level. We propose a pipeline for identifying abstractions for event generalizations from \texttt{GLUCOSE} \citep{mostafazadeh-etal-2020-glucose}, a large-scale dataset of implicit commonsense causal knowledge, from which we subsequently extract $1,4$K causal pairs. Our experiments highlight the ongoing challenges of using statistical methods and/or LLMs for automatic abstraction identification and causal discovery in NLP. Nonetheless, we demonstrate that the abstract causal knowledge provided in \texttt{ACCESS} can be leveraged for enhancing QA reasoning performance in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。