提出新架构Avey，用选相关词替代注意力，处理长文本更高效。
Don't Pay Attention
- 用排名器选关键词，再由神经处理器处理，跳过无关内容
- 在长序列任务中表现远超Transformer，短任务也持平
- 适合需要长距离依赖的文本处理，如文档摘要、问答
Transformer因其可并行训练和高效的自回归解码，已成为现代语言模型的标准。然而，其固定上下文窗口及自注意力机制带来的二次时间与内存开销仍是核心瓶颈。这促使人们重新关注线性扩展序列长度的循环架构，但牺牲了并行性。本文提出Avey，一种突破注意力与循环结构的新基础架构。Avey通过排名器与自回归神经处理器配对，仅选择并上下文化当前词最相关的词。它将序列长度与上下文宽度解耦，从而实现任意长度序列的有效高效处理。结果表明，Avey在多种标准短程NLP基准上表现优于或相当，而在需长距离依赖建模的任务中显著超越Transformer。
原文摘要 · Abstract (English)
The Transformer has become the de facto standard for modern language models owing to its parallelizable training and effective autoregressive decoding. However, its fixed context window and the quadratic time and memory costs of its self-attention mechanism remain central bottlenecks. These constraints have revived interest in recurrent architectures that scale linearly with sequence length, but at the cost of reduced parallelism. In this paper, we introduce Avey, a new foundational architecture that breaks away from both attention and recurrence. Avey pairs a ranker with an autoregressive neural processor to select and contextualize only the most relevant tokens for any given token. Specifically, it decouples sequence length from context width, thus enabling effective and efficient processing of arbitrarily long sequences. Results show that Avey compares favorably to the Transformer across a variety of standard short-range NLP benchmarks, while significantly outperforming it on tasks requiring long-range dependency modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。