用神经稳态机制提升Transformer稀疏性，性能优于传统模型
Homeostasis and Sparsity in Transformer
- 引入稳态机制RFB-kWTA与智能抑制，动态调节注意力与输出层
- 在Multi30K数据集上达0.3062 BLEU，优于基准模型0.2768和仅用丢弃的模型0.3007
- 适合关注模型稀疏性、类脑计算与注意力机制优化的研究者
Transformer架构已成为现代神经网络的核心，在文本生成、机器翻译、图像与音频处理等任务中发挥关键作用。本文受杰夫·霍金斯关于新皮质工作机制的启发，提出将稳态机制如RFB-kWTA与“智能抑制”引入Transformer的注意力模块及输出层，并在多个位置实验稀疏分布式表示。RFB-kWTA通过统计层激活时序分布，增强罕见激活值并抑制频繁激活；“智能抑制”则基于激活频率采样稀疏掩码，使更罕见的激活更易被保留。实验表明，所提方法在Multi30K数据集上达到0.3062 BLEU，显著优于经典Transformer（0.2768）及仅使用丢弃的模型（0.3007）。
原文摘要 · Abstract (English)
The transformer architecture has become an integral part of the field of modern neural networks, playing a crucial role in a variety of tasks, such as text generation, machine translation, image and audio processing, among others. There is also an alternative approach to building intelligent systems, proposed by Jeff Hawkins and inspired by the processes occurring in the neocortex. In our article we want to combine some of these ideas and to propose the use of homeostasis mechanisms, such as RFB-kWTA and "Smart" Inhibition, in the attention mechanism of the transformer and at the output of the transformer block, as well as conducting an experiment involving the introduction of sparse distributed representations of the transformer at various points. RFB-kWTA utilizes statistics of layer activations across time to adjust the entire layer, enhancing the values of rare activations while reducing those of frequent ones. "Smart" Inhibition also uses activation statistics to sample sparsity masks, with rarer activation times are more likely to be activated. Our proposed mechanisms significantly outperform the classical transformer 0.2768 BLEU and a model that only makes use of dropout in the attention mechanism and output of the transformer block 0.3007 BLEU, achieving a score of 0.3062 on the Multi30K dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。