揭示Transformer在上下文分类中的分层动态机制
Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classification
- 通过每层保持特征与标签置换等变性，使计算过程可识别
- 发现一种深度索引的递推更新规则，驱动类别分离增强
- 适合关注模型可解释性与几何算法动机的研究者
Transformer 能从少量标注样本中完成上下文分类，但推理阶段的算法仍不透明。本文研究多类线性分类在硬无间隔场景下，通过在每一层强制特征与标签置换等变性，使计算过程可识别。该方法在保持功能等价的同时，生成高度结构化的权重。由此提取出一个显式的深度索引递推关系：在 softmax Transformer 内部首次发现的端到端可识别的涌现更新规则。由混合特征-标签格拉姆结构形成的注意力矩阵，驱动训练样本、标签与测试探针的耦合更新。其动态实现了一种几何驱动的算法模式，可证明地增强类别分离，并带来鲁棒的期望类别对齐。
原文摘要 · Abstract (English)
Transformers can perform in-context classification from a few labeled examples, yet the inference-time algorithm remains opaque. We study multi-class linear classification in the hard no-margin regime and make the computation identifiable by enforcing feature- and label-permutation equivariance at every layer. This enables interpretability while maintaining functional equivalence and yields highly structured weights. From these models we extract an explicit depth-indexed recursion: an end-to-end identified, emergent update rule inside a softmax transformer, to our knowledge the first of its kind. Attention matrices formed from mixed feature-label Gram structure drive coupled updates of training points, labels, and the test probe. The resulting dynamics implement a geometry-driven algorithmic motif, which can provably amplify class separation and yields robust expected class alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。