模仿人类眼动机制，用注意力网络压缩输入序列，大幅减少计算量。
Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes
- 用预训练模型学习关键区域，仅保留重要特征作为输入序列。
- 计算量减少近80%，性能与原模型相当。
- 适合需要轻量化部署的视觉任务，如移动端推理。
Transformer网络受限于注意力矩阵的二次复杂度，导致序列长度难以扩展。传统自注意力会考虑整个序列，但实际关注区域是稀疏的。人类在观察图像或场景时采用称为眼动（saccades）的稀疏注意力机制，聚焦关键特征可显著降低计算时间。本文提出一种「眼动注意力网络」（Saccade Attention Network），通过大型预训练模型学习应关注的位置，并用于预处理图像，将输入序列缩减至仅关键特征。实验表明，该方法可将计算量减少近80%，同时保持与原始模型相近的性能表现。
原文摘要 · Abstract (English)
One of the limitations of transformer networks is the sequence length due to the quadratic nature of the attention matrix. Classical self attention uses the entire sequence length, however, the actual attention being used is sparse. Humans use a form of sparse attention when analyzing an image or scene called saccades. Focusing on key features greatly reduces computation time. By using a network (Saccade Attention Network) to learn where to attend from a large pre-trained model, we can use it to pre-process images and greatly reduce network size by reducing the input sequence length to just the key features being attended to. Our results indicate that you can reduce calculations by close to 80% and produce similar results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。