用新模型提升因果发现中梯度估计精度,让大模型无需微调也能更好推理因果关系。
Score-informed Neural Operator for Enhancing Ordering-based Causal Discovery
- 设计神经算子模型SciNO,稳定估算对数密度的海森对角线。
- 在合成与真实数据上使排序偏差降低42.7%和31.5%,内存效率高。
- 可直接增强大模型因果推理能力,无需额外训练或提示工程。
基于顺序的因果发现方法通过识别因果图的拓扑顺序,提供可扩展的替代方案以避免组合搜索。在加性噪声模型(ANM)假设下,近期基于分数匹配的因果顺序方法依赖于对对数密度海森对角线的准确估计。本文旨在改进该海森对角线的近似,从而提升基于顺序的因果发现算法性能。现有依赖斯坦因梯度估计的方法计算开销大、内存占用高,而基于扩散模型的方法因得分模型的二阶导数仍不稳定。为此,我们提出评分信息神经算子(SciNO),一种定义在光滑函数空间中的概率生成模型,能稳定近似海森对角线并保留得分建模中的结构信息。实验证明,相较DiffAN,SciNO在合成图上平均降低42.7%的顺序偏差,在真实数据集上降低31.5%;同时保持内存效率与可扩展性。此外,我们提出一种概率控制算法,将SciNO的概率估计与自回归模型先验结合,实现基于语义信息的可靠数据驱动因果排序,从而在不进行额外微调或提示工程的情况下,增强大语言模型的因果推理能力。
原文摘要 · Abstract (English)
Ordering-based approaches to causal discovery identify topological orders of causal graphs, providing scalable alternatives to combinatorial search methods. Under the Additive Noise Model (ANM) assumption, recent causal ordering methods based on score matching require an accurate estimation of the Hessian diagonal of the log-densities. In this paper, we aim to improve the approximation of the Hessian diagonal of the log-densities, thereby enhancing the performance of ordering-based causal discovery algorithms. Existing approaches that rely on Stein gradient estimators are computationally expensive and memory-intensive, while diffusion-model-based methods remain unstable due to the second-order derivatives of score models. To alleviate these problems, we propose Score-informed Neural Operator (SciNO), a probabilistic generative model in smooth function spaces designed to stably approximate the Hessian diagonal and to preserve structural information during the score modeling. Empirical results show that SciNO reduces order divergence by 42.7% on synthetic graphs and by 31.5% on real-world datasets on average compared to DiffAN, while maintaining memory efficiency and scalability. Furthermore, we propose a probabilistic control algorithm for causal reasoning with autoregressive models that integrates SciNO's probability estimates with autoregressive model priors, enabling reliable data-driven causal ordering informed by semantic information. Consequently, the proposed method enhances causal reasoning abilities of LLMs without additional fine-tuning or prompt engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。