arXiv:2511.16016cs.LGcs.AI2025-11被引 2

让大模型学会因果推理,用外部算法输出训练它更准地发现因果关系。

CARE: Turning LLMs Into Causal Reasoning Expert

  • 用成熟因果发现算法的输出作为提示,教大模型理解数据结构。
  • 微调后的小模型在因果推理上超越千倍参数的大模型。
  • 适合想提升模型因果能力的研究者或工业应用开发者。

大语言模型在推理与生成任务中表现优异,但缺乏识别因果关系的能力,这是人类智能的核心。我们发现,当要求大模型进行因果发现时,它们主要依赖变量名的语义,忽略观测数据。这在意料之中,因为大模型从未经过结构化数据训练。为此,我们尝试用成熟因果发现算法的输出作为提示,结果意外发现性能反而下降。为解决此问题,我们提出CARE框架,通过监督微调,教会大模型有效利用这些算法输出。实验表明,经CARE微调的Qwen2.5-1.5B模型在因果推理上显著优于传统算法和参数量超千倍的先进大模型,充分结合了自身知识与外部算法线索。

原文摘要 · Abstract (English)

Large language models (LLMs) have recently demonstrated impressive capabilities across a range of reasoning and generation tasks. However, research studies have shown that LLMs lack the ability to identify causal relationships, a fundamental cornerstone of human intelligence. We first conduct an exploratory investigation of LLMs' behavior when asked to perform a causal-discovery task and find that they mostly rely on the semantic meaning of variable names, ignoring the observation data. This is unsurprising, given that LLMs were never trained to process structural datasets. To first tackle this challenge, we prompt the LLMs with the outputs of established causal discovery algorithms designed for observational datasets. These algorithm outputs effectively serve as the sufficient statistics of the observation data. However, quite surprisingly, we find that prompting the LLMs with these sufficient statistics decreases the LLMs' performance in causal discovery. To address this current limitation, we propose CARE, a framework that enhances LLMs' causal-reasoning ability by teaching them to effectively utilize the outputs of established causal-discovery algorithms through supervised fine-tuning. Experimental results show that a finetuned Qwen2.5-1.5B model produced by CARE significantly outperforms both traditional causal-discovery algorithms and state-of-the-art LLMs with over a thousand times more parameters, demonstrating effective utilization of its own knowledge and the external algorithmic clues.

因果推理大模型微调算法提示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。