Transformer通过高效模拟局部多项式估计，实现最优非参数回归。
Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers
- 用核加权多项式基逼近局部多项式估计
- 在 $n^{-2α/(2α+d)}$ 误差下达到最优收敛率
- 参数量与预训练序列远少于已有方法
研究了在 $α$-Hölder 光滑回归函数下，基于 $n$ 个上下文样本和 $d$ 维协变量的上下文学习。我们证明，具有 $Θ(\log n)$ 参数和 $Ω\bigl(n^{2α/(2α+d)}\log^3 n\bigr)$ 预训练序列的预训练 Transformer，可实现均方误差 $O\bigl(n^{-2α/(2α+d)}\bigr)$ 的极小极大最优收敛率。该结果所需参数量和预训练序列显著少于以往工作。核心在于证明 Transformer 能通过实现核加权多项式基并运行梯度下降，高效逼近局部多项式估计。
原文摘要 · Abstract (English)
We study in-context learning for nonparametric regression with $α$-Hölder smooth regression functions, for some $α>0$. We prove that, with $n$ in-context examples and $d$-dimensional regression covariates, a pretrained transformer with $Θ(\log n)$ parameters and $Ω\bigl(n^{2α/(2α+d)}\log^3 n\bigr)$ pretraining sequences can achieve the minimax optimal rate of convergence $O\bigl(n^{-2α/(2α+d)}\bigr)$ in mean squared error. Our result requires substantially fewer transformer parameters and pretraining sequences than previous results in the literature. This is achieved by showing that transformers are able to approximate local polynomial estimators efficiently by implementing a kernel-weighted polynomial basis and then running gradient descent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。