RNN新架构在长短文本上都表现优异，还支持高效训练和推理。
Language Modeling With Factorization Memory
- 基于Mamba-2改进，训练时可并行计算，推理时资源恒定
- 稀疏更新部分状态，效率提升但性能几乎不变
- 适合追求高效长文本建模的开发者或研究者
我们提出因子分解记忆（Factorization Memory），一种高效的循环神经网络（RNN）架构，在短上下文语言建模任务上表现接近于Transformer模型，并在长上下文场景中展现出更优的泛化能力。该模型基于Mamba-2，使因子分解记忆在训练时能够利用并行计算，同时在推理过程中保持恒定的计算与内存复杂度。为进一步优化模型效率与表征能力，我们开发了因子分解记忆的稀疏形式，仅在每一步更新部分循环状态，同时保留其稠密版本的强大性能。据我们所知，这是首个成功结合稀疏记忆激活与短长上下文均具竞争力表现的RNN架构。本工作对因子分解记忆与Transformer及Mamba-2架构进行了系统的实证分析。
原文摘要 · Abstract (English)
We propose Factorization Memory, an efficient recurrent neural network (RNN) architecture that achieves performance comparable to Transformer models on short-context language modeling tasks while also demonstrating superior generalization in long-context scenarios. Our model builds upon Mamba-2, enabling Factorization Memory to exploit parallel computations during training while preserving constant computational and memory complexity during inference. To further optimize model efficiency and representational capacity, we develop a sparse formulation of Factorization Memory that updates only a subset of recurrent states at each step while preserving the strong performance of its dense counterpart. To our knowledge, this represents the first RNN architecture that successfully combines sparse memory activation with competitive performance across both short and long-context settings. This work provides a systematic empirical analysis of Factorization Memory in comparison to Transformer and Mamba-2 architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。