arXiv:2604.04250cs.CL2026-04

用连续波模型替代注意力,实现超长文本高效建模。

CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling

  • 用复数相位累积实现线性时间序列混合,避免传统注意力的二次开销。
  • 在200万词上下文中保持8.72GB显存,突破传统模型内存瓶颈。
  • 适合需要长上下文处理的场景,如文档理解、连续推理任务。

现代大语言模型依赖于二次增长的Transformer自注意力机制。近期线性时间替代方案(如状态空间模型)常在长上下文中出现信号退化。本文提出连续声波网络(CAWN),一种全连续序列混合架构。不同于离散矩阵注意力,CAWN将隐藏状态投影为多头复数域相位矢量,通过因果 $O(L)$ 相位累积实现序列混合。为防止超长上下文中的信号退化,引入双门控选择性相位共振机制,包含频率相关保留、直通估计硬阈值门控及时间语法缓存以捕捉短期局部依赖。标准密集线性投影被深度调制卷积替代,实现最优空间频率混合,并辅以块注意力残差进行深度状态路由。模型扩展至1.5亿参数,采用定制Triton内核,在float32下实现硬件高效的真复数相位累积。在1000亿词语料的连续流训练中,于50亿词里程碑处评估。靶向语义检索协议验证显示,模型具备稳健的词汇获取能力与显式上下文去噪学习。通过分块预填充实现 $O(1)$ 状态传递,模型可在200万词范围内准确检索目标信息,峰值显存严格维持在8.72 GB,实证突破 $O(L^2)$ 上下文内存墙。

原文摘要 · Abstract (English)

Modern Large Language Models (LLMs) rely on Transformer self-attention, which scales quadratically with sequence length. Recent linear-time alternatives, like State Space Models (SSMs), often suffer from signal degradation over extended contexts. We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture. Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors, achieving sequence mixing through a causal, $O(L)$ Phase Accumulation mechanism. To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism incorporating Frequency-Dependent Retention, Hard-Threshold Gating via Straight-Through Estimation, and a Temporal Syntax Cache to capture short-term local dependencies. We also replace standard dense linear projections with Depth-wise Harmonic Convolutions for optimal spatial frequency mixing, augmented by Block Attention Residuals for depth-wise state routing. Scaled to a 150M-parameter model, CAWN utilizes custom Triton kernels for hardware-efficient, true-complex phase accumulation in float32. Trained via a continuous streaming loop on a 100-Billion-token corpus, the prototype is evaluated at a 5-Billion-token milestone. Empirical evaluations via a Targeted Semantic Retrieval protocol demonstrate robust vocabulary acquisition and extended explicitly learned contextual denoising. By leveraging $O(1)$ state-passing via chunked prefill, the model retrieves targeted information across 2,000,000 tokens while strictly plateauing at 8.72 GB of Peak VRAM, empirically overcoming the $O(L^2)$ context memory wall.

语言模型连续建模长文本低显存

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。