模型训练时自动决定结构,不用人为设定层数和注意力头数。
DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations)
- 用原型向量替代传统前馈层,通过训练自发分离特征方向。
- 按需动态增加注意力头,新头触发条件由未捕获信息量决定。
- 理论证明结构唯一且最小,适合追求高效部署的研究者。
Transformer类神经网络需在训练前确定注意力头数、深度和宽度,但这些选择缺乏任务信息,导致模型普遍过大。本文提出DDCL-INCRT,一种训练中自组织的Transformer架构。其核心为两项互补机制:一是基于深度双竞争学习(DDCL)的原型字典,以学习数据中最富信息的方向,原型间自动分离而无需显式正则化;二是增量式变换器(INCRT),从一个注意力头开始,仅当现有头无法捕捉的方向性信息超过阈值时才添加新头。理论证明,二者相互强化:新增头促进原型分离,而分离后的信号又提升下一次添加的触发概率。收敛后,网络形成按表征粒度排序的层次化头结构,该结构被证明在给定条件下是唯一且最小的最优解。全文建立了稳定性、收敛性和剪枝安全性的形式化保证。
原文摘要 · Abstract (English)
Modern neural networks of the transformer family require the practitioner to decide, before training begins, how many attention heads to use, how deep the network should be, and how wide each component should be. These decisions are made without knowledge of the task, producing architectures that are systematically larger than necessary: empirical studies find that a substantial fraction of heads and layers can be removed after training without performance loss. This paper introduces DDCL-INCRT, an architecture that determines its own structure during training. Two complementary ideas are combined. The first, DDCL (Deep Dual Competitive Learning), replaces the feedforward block with a dictionary of learned prototype vectors representing the most informative directions in the data. The prototypes spread apart automatically, driven by the training objective, without explicit regularisation. The second, INCRT (Incremental Transformer), controls the number of heads: starting from one, it adds a new head only when the directional information uncaptured by existing heads exceeds a threshold. The main theoretical finding is that these two mechanisms reinforce each other: each new head amplifies prototype separation, which in turn raises the signal triggering the next addition. At convergence, the network self-organises into a hierarchy of heads ordered by representational granularity. This hierarchical structure is proved to be unique and minimal, the smallest architecture sufficient for the task, under the stated conditions. Formal guarantees of stability, convergence, and pruning safety are established throughout. The architecture is not something one designs. It is something one derives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。