提出新型注意力计算方法，实现高效长序列建模。
Myosotis: structured computation for attention like layer
- 基于树状矩阵的高效求逆机制，重构注意力计算结构
- 在保持序列长度线性复杂度的同时，优于传统稀疏与递归方法
- 适合长序列建模任务，如基因序列、文档处理
注意力层对输入元素间的成对交互进行建模，但无结构性假设时，内存和计算量随序列长度呈平方增长。现有缓解方式为引入稀疏性或沿序列建立递归依赖（如SSM），但各有缺陷。本文提出一种新算法，结合两者优势，基于树状矩阵的高效求逆机制重构注意力计算，实现可扩展的序列建模。
原文摘要 · Abstract (English)
Attention layers apply a sequence-to-sequence mapping whose parameters depend on the pairwise interactions of the input elements. However, without any structural assumptions, memory and compute scale quadratically with the sequence length. The two main ways to mitigate this are to introduce sparsity by ignoring a sufficient amount of pairwise interactions or to introduce recurrent dependence along them, as SSM does. Although both approaches are reasonable, they both have disadvantages. We propose a novel algorithm that combines the advantages of both concepts. Our idea is based on the efficient inversion of tree-structured matrices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。