证明Transformer能通过梯度下降学习特定教师模型并实现最优泛化。
Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models
- 用简化位置注意力的单层Transformer学习多种教师模型。
- 可恢复教师模型所有参数块，达到最优总体损失。
- 在弱假设下对分布外数据仍有良好泛化能力，适合理论研究者。
Transformer在众多应用中取得巨大成功，但其理论基础仍不明确。本文从理论上研究将Transformer作为学生模型，学习一类教师模型的能力。该类教师模型包括含平均池化的卷积层、图卷积层，以及多种经典统计学习模型，如稀疏标记选择模型变体和组稀疏线性预测器。当学习这类教师模型时，我们证明：采用简化“仅位置”注意力的一层Transformer，能够成功恢复教师模型的所有参数块，从而实现最优总体损失。基于训练后的Transformer对教师模型的有效模仿，进一步表明其在温和假设下，可在广泛分布外数据上实现良好泛化。分析的核心在于识别不同学习任务共有的基本双线性结构，使我们能为这些任务统一建立学习保证。
原文摘要 · Abstract (English)
Transformers have achieved great success across a wide range of applications, yet the theoretical foundations underlying their success remain largely unexplored. To demystify the strong capacities of transformers applied to versatile scenarios and tasks, we theoretically investigate utilizing transformers as students to learn from a class of teacher models. Specifically, the teacher models covered in our analysis include convolution layers with average pooling, graph convolution layers, and various classic statistical learning models, including a variant of sparse token selection models [Sanford et al., 2023, Wang et al., 2024] and group-sparse linear predictors [Zhang et al., 2025]. When learning from this class of teacher models, we prove that one-layer transformers with simplified "position-only'' attention can successfully recover all parameter blocks of the teacher models, thus achieving the optimal population loss. Building upon the efficient mimicry of trained transformers towards teacher models, we further demonstrate that they can generalize well to a broad class of out-of-distribution data under mild assumptions. The key in our analysis is to identify a fundamental bilinear structure shared by various learning tasks, which enables us to establish unified learning guarantees for these tasks when treating them as teachers for transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。