测试13个大模型在生物医学等领域的因果关系识别能力
Benchmarking LLMs for Pairwise Causal Discovery in Biomedical and Multi-Domain Contexts
- 构建12个数据集的统一评测框架，测试模型发现和提取因果关系的能力
- 最佳模型检测准确率仅49.57%，提取准确率最高47.12%，表现普遍不佳
- 模型对隐含、跨句或多重因果关系处理能力弱，适合关注可信AI的研究者
为评估大语言模型在生物医学等高风险领域安全部署的因果推理能力，本文测试了13个开源LLMs在文本中进行成对因果发现（PCD）的表现。基于12个多样化数据集的基准测试，评估两项核心能力：因果检测（识别文本是否包含因果关系）与因果提取（准确识别原因和结果短语）。实验对比了零样本、思维链（CoT）及少量示例上下文学习（FICL）等多种提示方法。结果显示，当前模型存在明显缺陷：最佳检测模型DeepSeek-R1-Distill-Llama-70B的平均得分仅为49.57%（C_detect），最佳提取模型Qwen2.5-Coder-32B-Instruct的得分也仅达47.12%（C_extract）。模型在简单、明确、单句关系上表现较好，但在隐含关系、跨句链接或多对因果场景中性能急剧下降。研究采用经高一致性验证的数据集（κ≥0.758），并公开全部数据、代码与提示模板，以推动后续研究。
原文摘要 · Abstract (English)
The safe deployment of large language models (LLMs) in high-stakes fields like biomedicine, requires them to be able to reason about cause and effect. We investigate this ability by testing 13 open-source LLMs on a fundamental task: pairwise causal discovery (PCD) from text. Our benchmark, using 12 diverse datasets, evaluates two core skills: 1) \textbf{Causal Detection} (identifying if a text contains a causal link) and 2) \textbf{Causal Extraction} (pulling out the exact cause and effect phrases). We tested various prompting methods, from simple instructions (zero-shot) to more complex strategies like Chain-of-Thought (CoT) and Few-shot In-Context Learning (FICL). The results show major deficiencies in current models. The best model for detection, DeepSeek-R1-Distill-Llama-70B, only achieved a mean score of 49.57\% ($C_{detect}$), while the best for extraction, Qwen2.5-Coder-32B-Instruct, reached just 47.12\% ($C_{extract}$). Models performed best on simple, explicit, single-sentence relations. However, performance plummeted for more difficult (and realistic) cases, such as implicit relationships, links spanning multiple sentences, and texts containing multiple causal pairs. We provide a unified evaluation framework, built on a dataset validated with high inter-annotator agreement ($κ\ge 0.758$), and make all our data, code, and prompts publicly available to spur further research. \href{https://github.com/sydneyanuyah/CausalDiscovery}{Code available here: https://github.com/sydneyanuyah/CausalDiscovery}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。