让RNN同时具备高效与强表达能力，突破传统对角混合限制。
Fixed-Point RNNs: Interpolating from Diagonal to Dense
- 用可并行的对角RNN构造稠密RNN的不动点，实现高效计算。
- 在A5和S5任务上达到当前最优，复制等任务表现不降。
- 适合需要高精度状态追踪的序列建模场景。
线性循环神经网络（RNN）和状态空间模型（SSM，如Mamba）已成为Transformer中替代Softmax注意力的有前景方案。然而，现有模型因依赖通道独立（即对角）序列混合，未能体现RNN的完整状态追踪表达能力。本文研究了一类稠密线性RNN的参数化方式，将其表示为可并行的对角线性RNN的不动点。所得模型可在固定参数量下自然权衡表达力与效率，在状态追踪基准A5和S5上取得当前最佳表现，同时在复制等任务上保持相当性能。
原文摘要 · Abstract (English)
Linear recurrent neural networks (RNNs) and state-space models (SSMs) such as Mamba have become promising alternatives to softmax-attention as sequence mixing layers in Transformer architectures. Current models, however, do not exhibit the full state-tracking expressivity of RNNs because they rely on channel-wise (i.e. diagonal) sequence mixing. In this paper, we investigate parameterizations of a large class of dense linear RNNs as fixed-points of parallelizable diagonal linear RNNs. The resulting models can naturally trade expressivity for efficiency at a fixed number of parameters and achieve state-of-the-art results on the state-tracking benchmarks $A_5$ and $S_5$, while matching performance on copying and other tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。