Transformer可自适应解决线性回归中的内生性问题，效果优于传统方法。
Transformers Handle Endogeneity in In-Context Linear Regression
- 利用变压器架构模拟梯度优化，自动逼近两阶段最小二乘法解。
- 在内生性存在时，模型预测误差和系数估计偏差显著低于2SLS。
- 适合需要稳健回归分析的经济学、社会科学领域研究者。
我们探究了变压器在上下文线性回归中处理内生性问题的能力。核心发现是，变压器通过工具变量（IV）具备内在机制有效应对内生性。首先，我们证明变压器架构能模拟基于梯度的双层优化过程，并以指数速率收敛至广泛使用的两阶段最小二乘法（2SLS）解。其次，我们提出一种上下文预训练方案，并提供理论保证：预训练损失的全局最小值具有小的超额损失。大量实验验证了这些理论结果，表明训练后的变压器在内生性存在时，提供的上下文预测和系数估计比2SLS方法更具鲁棒性和可靠性。
原文摘要 · Abstract (English)
We explore the capability of transformers to address endogeneity in in-context linear regression. Our main finding is that transformers inherently possess a mechanism to handle endogeneity effectively using instrumental variables (IV). First, we demonstrate that the transformer architecture can emulate a gradient-based bi-level optimization procedure that converges to the widely used two-stage least squares $(\textsf{2SLS})$ solution at an exponential rate. Next, we propose an in-context pretraining scheme and provide theoretical guarantees showing that the global minimizer of the pre-training loss achieves a small excess loss. Our extensive experiments validate these theoretical findings, showing that the trained transformer provides more robust and reliable in-context predictions and coefficient estimates than the $\textsf{2SLS}$ method, in the presence of endogeneity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。