Transformer自回归训练可自动捕捉时间因果关系，无需额外目标。
Transformer Is Inherently a Causal Learner
- 通过梯度敏感性直接还原真实因果图，无需显式因果约束。
- 在非线性、长依赖等复杂场景下超越现有方法，数据越多越准。
- 适合想用大模型做因果发现或提升模型可解释性的研究者。
我们发现，以自回归方式训练的Transformer在学习过程中自然编码了时延因果结构。在多变量时间序列预测中，输出对过去输入的梯度敏感性可直接恢复底层因果图，无需任何显式因果目标或结构约束。我们在标准可识别条件下进行了理论证明，并提出基于聚合梯度归因的实用提取方法。在非线性动态、长期依赖和非平稳系统等挑战性场景下，该方法显著优于现有最优发现算法，尤其在数据异质性增加时表现更优，展现出随数据量与异质性增长而提升的可扩展性——这是传统方法所不具备的特性。这一统一视角为未来以基础模型为核心进行因果发现奠定了基础，同时使基础模型通过因果视角获得可解释性与性能增强。
原文摘要 · Abstract (English)
We reveal that transformers trained in an autoregressive manner naturally encode time-delayed causal structures in their learned representations. When predicting future values in multivariate time series, the gradient sensitivities of transformer outputs with respect to past inputs directly recover the underlying causal graph, without any explicit causal objectives or structural constraints. We prove this connection theoretically under standard identifiability conditions and develop a practical extraction method using aggregated gradient attributions. On challenging cases such as nonlinear dynamics, long-term dependencies, and non-stationary systems, this approach greatly surpasses the performance of state-of-the-art discovery algorithms, especially as data heterogeneity increases, exhibiting scaling potential where causal accuracy improves with data volume and heterogeneity, a property traditional methods lack. This unifying view lays the groundwork for a future paradigm where causal discovery operates through the lens of foundation models, and foundation models gain interpretability and enhancement through the lens of causality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。