用快速傅里叶变换实现高效长序列建模，替代注意力机制。
Caracal: Causal Architecture via Spectral Mixing
- 用快速傅里叶变换实现多头频域混合，复杂度降至O(L log L)
- 通过频域因果掩码实现自回归生成，突破傅里叶模型瓶颈
- 仅用标准库操作，部署兼容性好，适合通用场景
大语言模型在处理长序列时受限于注意力机制的二次计算开销和位置编码的局限性。为解决这些问题，我们提出Caracal，一种新型架构，用参数高效的O(L log L)多头傅里叶（MHF）模块替代注意力。主要贡献包括：(1) 利用快速傅里叶变换（FFT）进行序列混合，天然缓解上述两个瓶颈；(2) 提出频域因果掩码技术，通过非对称填充与截断实现自回归能力，克服了傅里叶基生成模型的关键障碍；(3) 与依赖硬件特定实现（如Mamba）的高效模型不同，Caracal仅使用标准库操作，确保良好可移植性，消除常见部署壁垒。评估表明，Caracal在性能上媲美Transformer与SSM基线，为高效长序列建模提供了一条简洁可扩展的路径。代码见附录。
原文摘要 · Abstract (English)
The scalability of Large Language Models to long sequences is hindered by the quadratic cost of attention and the limitations of positional encodings. To address these, we introduce Caracal, a novel architecture that replaces attention with a parameter-efficient, O(L log(L)) Multi-Head Fourier (MHF) module. Our contributions are threefold: (1) We leverage the Fast Fourier Transform (FFT) for sequence mixing, inherently addressing both bottlenecks mentioned above. (2) We apply a frequency-domain causal masking technique that enforces autoregressive capabilities via asymmetric padding and truncation, overcoming a critical barrier for Fourier-based generative models. (3) Unlike efficient models relying on hardware-specific implementations (e.g., Mamba), we uses standard library operators. This ensures robust portability, eliminating common deployment barriers. Evaluations demonstrate that Caracal performs competitively with Transformer and SSM baselines, offering a scalable and simple pathway for efficient long-sequence modeling. Code is available in Appendix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。