用注意力机制自动发现数据聚类结构，理论证明其有效性和动态适应能力。
Attention-based clustering
- 基于双头注意力层，无监督学习下可自动对齐高斯混合数据的中心点。
- 即使没有可训练参数，固定身份矩阵的注意力层仍能实现上下文量化。
- 适合研究自监督学习、聚类算法与模型内在机制的学者参考。
Transformer已成为一种强大的神经网络架构，适用于多种学习任务。本文从理论上分析其在无监督设置下从数据中自动提取结构的能力。特别地，我们证明当输入数据来自高斯混合模型时，Transformer适用于聚类任务。为此，我们研究了一个简化的双头注意力层，并定义了一个总体风险，其在无标签数据下的最小化可使头部参数对齐真实的混合中心。这一现象凸显了注意力层捕捉底层分布结构的能力。此外，我们进一步考察了将键、查询和值矩阵固定为恒等矩阵的注意力层，发现即便没有任何可训练参数，它仍能实现上下文量化，揭示了基于Transformer的方法对输入特定分布的惊人动态适应能力。
原文摘要 · Abstract (English)
Transformers have emerged as a powerful neural network architecture capable of tackling a wide range of learning tasks. In this work, we provide a theoretical analysis of their ability to automatically extract structure from data in an unsupervised setting. In particular, we demonstrate their suitability for clustering when the input data is generated from a Gaussian mixture model. To this end, we study a simplified two-head attention layer and define a population risk whose minimization with unlabeled data drives the head parameters to align with the true mixture centroids. This phenomenon highlights the ability of attention-based layers to capture underlying distributional structure. We further examine an attention layer with key, query, and value matrices fixed to the identity, and show that, even without any trainable parameters, it can perform in-context quantization, revealing the surprising capacity of transformer-based methods to adapt dynamically to input-specific distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。