新模型突破注意力机制瓶颈，可处理百万级长文本。
Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons
- 不用注意力机制，改用状态空间与多分辨率卷积
- 处理长达数十万至百万token文本，计算量近线性增长
- 适合超长文档分析、法律/医学文本等需要全局理解的场景
我们提出一种基于非注意力机制的大语言模型架构，可高效处理长达数十万甚至百万级标记的上下文。不同于传统Transformer因自注意力机制导致的二次方内存和计算开销，该模型完全避免了标记间的注意力计算。其核心由四个互补组件构成：受S4启发的状态空间块（学习连续时间卷积核，随序列长度近似线性扩展）、多分辨率卷积层（在不同扩张层级捕捉局部上下文）、轻量级循环监督器（维护跨分块的全局隐藏状态）以及检索增强的外部记忆（存储并检索高阶分块嵌入，不引入二次操作）。
原文摘要 · Abstract (English)
We present a novel non attention based architecture for large language models (LLMs) that efficiently handles very long context windows, on the order of hundreds of thousands to potentially millions of tokens. Unlike traditional Transformer designs, which suffer from quadratic memory and computation overload due to the nature of the self attention mechanism, our model avoids token to token attention entirely. Instead, it combines the following complementary components: State Space blocks (inspired by S4) that learn continuous time convolution kernels and scale near linearly with sequence length, Multi Resolution Convolution layers that capture local context at different dilation levels, a lightweight Recurrent Supervisor to maintain a global hidden state across sequential chunks, and Retrieval Augmented External Memory that stores and retrieves high-level chunk embeddings without reintroducing quadratic operations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。