提出新型注意力机制，让Transformer在流式数据中高效运行。
Continual Low-Rank Scaled Dot-product Attention
- 用Nyström近似重构注意力机制，支持持续推理
- 运算量降低三个数量级，性能不下降
- 适合资源受限的实时流数据处理场景
Transformer凭借捕捉序列数据关系的能力，在众多静态任务中取得成功，但其核心组件——缩放点积注意力的计算与内存开销常被忽视。这使得其在响应延迟、计算和内存资源受限的流数据处理应用中难以部署。已有工作通过低秩近似、注意力稀疏化和高效公式等方法降低Transformer计算成本。本文提出一种基于Nyström近似的新型缩放点积注意力，适用于持续推理。在在线音频分类和在线动作检测任务上的实验表明，所提方法相比原始Transformer可将运算量降低高达三个数量级，同时保持与竞争模型相当的预测性能。
原文摘要 · Abstract (English)
Transformers are widely used for their ability to capture data relations in sequence processing, with great success for a wide range of static tasks. However, the computational and memory footprint of their main component, i.e., the Scaled Dot-product Attention, is commonly overlooked. This makes their adoption in applications involving stream data processing with constraints in response latency, computational and memory resources infeasible. Some works have proposed methods to lower the computational cost of Transformers, i.e. low-rank approximations, sparsity in attention, and efficient formulations for Continual Inference. In this paper, we introduce a new formulation of the Scaled Dot-product Attention based on the Nyström approximation that is suitable for Continual Inference. In experiments on Online Audio Classification and Online Action Detection tasks, the proposed Continual Scaled Dot-product Attention can lower the number of operations by up to three orders of magnitude compared to the original Transformers while retaining the predictive performance of competing models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。