用物理共振机制替代注意力，提升长文本建模效率与速度
ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling
- 将词元序列视为受驱一维隐场，用阻尼共振器函数替代注意力计算
- 在32K上下文长度下，解码速度达标准Transformer的6.47倍，准确率61.31%
- 适合需要高效处理超长文本的场景，如法律、医学文档分析
当前语言模型以Transformer架构为主，依赖自注意力机制实现跨多种模态和上下文的数据建模。然而，Transformer及其传统变体（如RNN、CNN）在处理长上下文时往往效率下降。本文提出ResonatorLM，一种基于物理原理的新机制，取代注意力计算。该方法将词元序列视为单一受驱一维隐场，用阻尼共振器的因果函数替代注意力点积。我们在经典网络架构上实现ResonatorLM，并在标准长上下文建模任务中测试。在600万参数的小规模设置下，训练与预填充速度随序列长度增加而提升；在32K词元长度时，解码速度相较优化后的标准Transformer提升6.47倍，WikiText任务准确率达61.31%（对比基准55.32%）。
原文摘要 · Abstract (English)
Contemporary language models are dominated by the transformer architecture, which leverages self-attention mechanisms to enable more efficient, parallelized training across a wide set of documents and corpora. This has allowed transformers to effectively model data across a wide range of modalities and contexts. However, transformers, along with their conventional counterparts such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), often struggle to maintain efficiency when processing long contexts. We introduce ResonatorLM, a new mechanism that replaces attention with a physics-derived alternative. ResonatorLM treats token sequences as a single, driven one-dimensional latent field and replaces attention dot products with causal functions of damped resonators. We implement ResonatorLM on a traditional network architecture and test it on standard long-context modeling tasks. We find that in a small, 6M matched setting, training and prefill speedups increase with sequence length, decode speed reaches 6.47x compared to that of a standard, optimized transformer at 32K tokens, and accuracy reaches 61.31 percent (compared to 55.32 percent) on WikiText.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。