让Transformer学会区分因果与相关，提升模型在分布外场景的可靠性。
OrthoFormer: Instrumental Variable Estimation in Transformer Hidden States via Neural Control Functions
- 通过神经控制函数在Transformer中直接嵌入工具变量估计，实现因果建模。
- 理论证明其偏差严格小于OLS，且随迭代次数几何级收敛，性能更优。
- 适合需要高鲁棒性、可解释性的序列决策系统，如医疗、金融等场景。
Transformer架构在序列建模上表现卓越，但本质上受限于相关性学习——它们捕捉由潜在混淆因子引起的虚假关联，而非不变的因果机制。我们将其视为一种认识论挑战：标准Transformer将静态背景因素（内在身份、风格、上下文）与动态因果流（状态演化、机制）混为一谈，导致分布外性能灾难性崩溃。为此，我们提出OrthoFormer，一种基于因果原理的架构，在Transformer模块中直接嵌入工具变量估计，利用神经控制函数实现因果推断。该框架建立在四个理论支柱之上：结构方向性（时间箭头约束）、表示正交性（潜在噪声分离）、因果稀疏性（马尔可夫毯近似）和端到端一致性（梯度解耦阶段分离）。我们证明，OrthoFormer在任意有效工具变量滞后下，偏差严格小于普通最小二乘法（OLS），残差偏差以几何速率衰减，形式为O(ρ^k)。我们揭示了自工具化中的偏差-方差-外生性三难困境，并识别出‘神经禁止回归’现象——移除梯度解耦虽能降低预测损失，却破坏因果有效性。实验验证了所有理论预测。OrthoFormer标志着从相关性到因果性序列建模的范式转变，对鲁棒性、可解释性及分布漂移下的可靠决策具有深远影响。
原文摘要 · Abstract (English)
Transformer architectures excel at sequential modeling yet remain fundamentally limited by correlational learning - they capture spurious associations induced by latent confounders rather than invariant causal mechanisms. We identify this as an epistemological challenge: standard Transformers conflate static background factors (intrinsic identity, style, context) with dynamic causal flows (state evolution, mechanism), leading to catastrophic out-of-distribution failure. We propose OrthoFormer, a causally grounded architecture that embeds instrumental variable estimation directly into Transformer blocks via neural control functions. Our framework rests on four theoretical pillars: Structural Directionality (time-arrow enforcement), Representation Orthogonality (latent-noise separation), Causal Sparsity (Markov Blanket approximation), and End-to-End Consistency (gradient- detached stage separation). We prove that OrthoFormer achieves bias strictly less than OLS for any valid instrument lag, with residual bias decaying geometrically as O(\r{ho}k ). We characterize the bias-variance-exogeneity trilemma inherent in self-instrumenting and identify the neural forbidden regression - where removing gradient detachment improves prediction loss while destroying causal validity. Experiments confirm all theoretical predictions. OrthoFormer represents a paradigm shift from correlational to causal sequence modeling, with implications for robustness, interpretability, and reliable decision-making under distribution shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。