用小模型模拟大模型，高效处理长序列输入。
Two Heads Are Better than One: Simulating Large Transformers with Small Ones
- 用多个短序列小模型并行模拟长序列大模型。
- 最坏情况下需 (N/M)² 个小模型，但多数场景只需 O(N/M) 个。
- 适合在资源受限设备上运行长序列Transformer任务。
自注意力的二次复杂度限制了Transformer在长输入序列上的扩展。而现代GPU等硬件对短序列Transformer的训练与推理高度优化。本文提出：能否利用小模型的效率来处理长序列？我们证明，任意长度为 $N$ 的大Transformer可由 $O((N/M)^2)$ 个长度为 $M \\< N$ 的小Transformer高效模拟，且该下界不可改进。然而，在平均情况输入、滑动窗口掩码及注意力下沉等自然场景中，仅需 $O(N/M)$ 个小模型即可实现最优模拟。
原文摘要 · Abstract (English)
The quadratic complexity of self-attention prevents transformers from scaling effectively to long input sequences. On the other hand, modern GPUs and other specialized hardware accelerators are well-optimized for processing small input sequences in transformers during both training and inference. A natural question arises: can we take advantage of the efficiency of small transformers to deal with long input sequences? In this paper, we show that transformers with long input sequences (large transformers) can be efficiently simulated by transformers that can only take short input sequences (small transformers). Specifically, we prove that any transformer with input length $N$ can be efficiently simulated by only $O((N/M)^2)$ transformers with input length $M \ll N$, and that this cannot be improved in the worst case. However, we then prove that in various natural scenarios including average-case inputs, sliding window masking and attention sinks, the optimal number $O(N/M)$ of small transformers suffice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。