用新型注意力机制压缩Transformer，提升效率且保持性能。
InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer
- 用曼哈顿距离和ReLU替代传统注意力的矩阵乘法
- 在GLUE和情感分析任务上表现接近原模型
- 适合追求低延迟、低功耗推理的NLP应用
本文探索通过融合模型压缩技术与抑制器注意力（inhibitor attention）来优化基于Transformer的语言模型。抑制器注意力采用曼哈顿距离和ReLU激活函数，取代传统的缩放点积注意力中的矩阵乘法与Softmax激活，有望在保持模型效果的同时降低计算量和能耗。我们对抑制器机制进行了改进以提升训练效率，并在DistilBERT架构上评估其性能。知识蒸馏实验表明，优化后的抑制器Transformer模型在标准NLP基准测试（包括通用语言理解评估GLUE和情感分析任务）中可达到具有竞争力的表现。
原文摘要 · Abstract (English)
This work explores optimizing transformer-based language models by integrating model compression techniques with inhibitor attention, a novel alternative attention mechanism. Inhibitor attention employs Manhattan distances and ReLU activations instead of the matrix multiplications and softmax activation of the conventional scaled dot-product attention. This shift offers potential computational and energy savings while maintaining model effectiveness. We propose further adjustments to improve the inhibitor mechanism's training efficiency and evaluate its performance on the DistilBERT architecture. Our knowledge distillation experiments indicate that the modified inhibitor transformer model can achieve competitive performance on standard NLP benchmarks, including General Language Understanding Evaluation (GLUE) and sentiment analysis tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。