Transformer在非理想数据下仍能稳定完成上下文回归，表现优于传统方法。
Transformers Learn Robust In-Context Regression under Distributional Uncertainty
- 用Transformer直接从提示中学习线性回归，不依赖高斯假设。
- 在非高斯系数、重尾噪声和非独立提示下，性能均超越经典基准。
- 适合研究分布不确定性下的模型自适应，尤其对实际应用有参考价值。
已有研究显示，在独立同分布、高斯噪声和高斯系数等严格假设下，Transformer可实现上下文学习进行线性回归。但真实数据常违背这些假设：输入、噪声和系数分布未知、非高斯，且提示间存在依赖。本文研究在广泛分布偏移下（包括非高斯系数、重尾噪声、非独立提示）的噪声线性回归的上下文学习。将Transformer与在对应最大似然准则下最优或次优的经典基线进行比较。在所有设置中，Transformer始终匹配或超越这些基线，证明其在现实分布不确定性下具有鲁棒的上下文适应能力，超越经典估计器。
原文摘要 · Abstract (English)
Recent work has shown that Transformers can perform in-context learning for linear regression under restrictive assumptions, including i.i.d. data, Gaussian noise, and Gaussian regression coefficients. However, real-world data often violate these assumptions: the distributions of inputs, noise, and coefficients are typically unknown, non-Gaussian, and may exhibit dependency across the prompt. This raises a fundamental question: can Transformers learn effectively in-context under realistic distributional uncertainty? We study in-context learning for noisy linear regression under a broad range of distributional shifts, including non-Gaussian coefficients, heavy-tailed noise, and non-i.i.d. prompts. We compare Transformers against classical baselines that are optimal or suboptimal under the corresponding maximum-likelihood criteria. Across all settings, Transformers consistently match or outperform these baselines, demonstrating robust in-context adaptation beyond classical estimators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。