通过增加注意力头数，减少模型深度，实现更小体积但不降精度的Transformer。
Leaner Transformers: More Heads, Less Depth
- 用更多注意力头改善注意力块条件，替代加深网络来提升性能。
- 在多个任务上减少30%-50%参数量，准确率保持不变。
- 适合追求高效部署的工程师和资源受限场景的研究者。
Transformer 通过注意力机制在大规模数据中捕捉复杂模式，推动了机器学习的发展，也催生了“越大越好”的认知。本文挑战这一观念，发现现有Transformer可能存在过度设计。我们揭示了一个理论原理：多头注意力的核心作用在于改善注意力块的条件性。基于此，我们重构主流架构，增加头数、减少深度。实践证明，条件性显著提升，使模型深度可大幅降低，参数量减少30%-50%，同时保持准确率。该方法在计算机视觉（ImageNet-1k）及语言与序列建模任务（GLUE、TinyStories、Long-Range Arena）中均表现稳定有效。
原文摘要 · Abstract (English)
Transformers have reshaped machine learning by utilizing attention mechanisms to capture complex patterns in large datasets, leading to significant improvements in performance. This success has contributed to the belief that "bigger means better", leading to ever-increasing model sizes. This paper challenge this ideology by showing that many existing transformers might be unnecessarily oversized. We discover a theoretical principle that redefines the role of multi-head attention. An important benefit of the multiple heads is in improving the conditioning of the attention block. We exploit this theoretical insight and redesign popular architectures with an increased number of heads. The improvement in the conditioning proves so significant in practice that model depth can be decreased, reducing the parameter count by up to 30-50% while maintaining accuracy. We obtain consistent benefits across a variety of transformer-based architectures of various scales, on tasks in computer vision (ImageNet-1k) as well as language and sequence modeling (GLUE benchmark, TinyStories, and the Long-Range Arena benchmark).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。