arXiv:2604.10703cs.LGcs.NE2026-04被引 3

模型训练时自动增删注意力头,实现结构自适应与高效压缩。

INCRT: An Incremental Transformer That Determines Its Own Architecture

  • 基于任务方向性动态增减注意力头,无需预设架构
  • 最终模型头数与理论预测值偏差小于12%,参数量减少3-7倍
  • 适合追求轻量化、高效率的NLP任务,无需预训练

Transformer架构通常依赖试错设计:注意力头数、深度和头大小在训练前固定,缺乏数学指导,导致系统性冗余——训练后半数至五分之四的头可移除而性能不变。本文提出INCRT(增量变压器),一种在训练中自主决定结构的架构。从单个头开始,当当前配置被证明不足时,每次添加一个头;当头变得冗余时则进行剪枝。每一步增长由一个在线计算的几何量驱动,无需验证阶段或人工调度。两个定理构成理论基础:其一(稳态收敛)表明系统总能到达有限停止状态,既最小化(无冗余头)又充分(捕获所有高于阈值的方向能量);其二(压缩感知类比)给出了头数的几何上界,取决于任务的谱复杂度。在SARS-CoV-2变体分类和SST-2情感分析实验中验证:预测与实际头数误差在12%以内,最终模型在特定任务上表现媲美BERT-base,但参数量减少3至7倍,且无需预训练。

原文摘要 · Abstract (English)

Transformer architectures are designed by trial and error: the number of attention heads, the depth, and the head size are fixed before training begins, with no mathematical principle to guide the choice. The result is systematic structural redundancy -- between half and four-fifths of all heads in a trained model can be removed without measurable loss -- because the architecture allocates capacity without reference to the actual requirements of the task.This paper introduces INCRT (Incremental Transformer), an architecture that determines its own structure during training. Starting from a single head, INCRT adds one attention head at a time whenever its current configuration is provably insufficient, and prunes heads that have become redundant. Each growth decision is driven by a single, online-computable geometric quantity derived from the task's directional structure, requiring no separate validation phase and no hand-tuned schedule. Two theorems form the theoretical backbone. The first (homeostatic convergence) establishes that the system always reaches a finite stopping configuration that is simultaneously minimal (no redundant heads) and sufficient (no uncaptured directional energy above the threshold). The second (compressed-sensing analogy) provides a geometric upper bound on the number of heads that this configuration can contain, as a function of the spectral complexity of the task. Experiments on SARS-CoV-2 variant classification and SST-2 sentiment analysis confirm both results: the predicted and observed head counts agree within 12% across all benchmarks, and the final architectures match or exceed BERT-base on distribution-specific tasks while using between three and seven times fewer parameters and no pre-training.

Transformer结构自适应轻量化注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。