Transformer能自动选对分组变量，实现高效分类。
Transformer Learns Optimal Variable Selection in Group-Sparse Classification
- 用单层Transformer+梯度下降学习分组稀疏模型
- 可准确识别关键变量组，忽略无关变量
- 预训练后小样本也能快速适应新任务
Transformer在众多应用中表现卓越，但其理论机制尚不明确。本文研究了如何通过训练使Transformer学习经典的具有“分组稀疏性”的统计模型：输入变量分为多个组，标签仅依赖于其中一组变量。理论上证明，经过梯度下降训练的单层Transformer能有效利用注意力机制，选择相关变量，忽略无关变量，实现准确分类。此外，经过良好预训练的单层Transformer可在少量样本下快速适配新下游任务，获得良好预测性能。本研究揭示了Transformer如何有效学习结构化数据。
原文摘要 · Abstract (English)
Transformers have demonstrated remarkable success across various applications. However, the success of transformers have not been understood in theory. In this work, we give a case study of how transformers can be trained to learn a classic statistical model with "group sparsity", where the input variables form multiple groups, and the label only depends on the variables from one of the groups. We theoretically demonstrate that, a one-layer transformer trained by gradient descent can correctly leverage the attention mechanism to select variables, disregarding irrelevant ones and focusing on those beneficial for classification. We also demonstrate that a well-pretrained one-layer transformer can be adapted to new downstream tasks to achieve good prediction accuracy with a limited number of samples. Our study sheds light on how transformers effectively learn structured data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。