隐式状态空间模型让RNN兼具并行训练与强表达力，突破传统瓶颈。
Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
- 通过迭代到固定点实现类RNN非线性状态转移
- 仅部分词元需完全收敛，支持大规模并行训练
- 在正则语言和自然语言任务上超越显式模型
状态空间模型（SSMs）与变换器主导语言建模领域，但其计算复杂度低于经典循环神经网络（RNNs），限制了表达能力。相反，RNN缺乏训练并行性，引发并行性与表达力之间的根本权衡。本文提出隐式SSMs，通过迭代变换直至收敛至固定点实现。理论上，隐式SSMs可模拟RNN的非线性状态转移；实证发现，近似收敛即足够，由此设计出可扩展的训练课程，仅少数词元需完全收敛，其余过程保持高度并行。该方法在正则语言任务中展现出卓越的状态跟踪能力，优于变换器和显式SSMs。进一步将隐式SSMs扩展至自然语言推理与大规模预训练，最大达13亿参数、2070亿词元，为迄今最大规模的隐式模型训练。显著的是，其在标准基准测试中优于对应显式模型。代码已公开于http://github.com/microsoft/implicit_languagemodels。
原文摘要 · Abstract (English)
State-space models (SSMs) and transformers dominate the language modeling landscape. However, they are constrained to a lower computational complexity than classical recurrent neural networks (RNNs), limiting their expressivity. In contrast, RNNs lack parallelization during training, raising fundamental questions about the trade off between parallelization and expressivity. We propose implicit SSMs, which iterate a transformation until convergence to a fixed point. Theoretically, we show that implicit SSMs implement the non-linear state-transitions of RNNs. Empirically, we find that only approximate fixed-point convergence suffices, enabling the design of a scalable training curriculum that largely retains parallelization, with full convergence required only for a small subset of tokens. Our approach demonstrates superior state-tracking capabilities on regular languages, surpassing transformers and SSMs. We further scale implicit SSMs to natural language reasoning tasks and pretraining of large-scale language models up to 1.3B parameters on 207B tokens representing, to our knowledge, the largest implicit model trained to date. Notably, our implicit models outperform their explicit counterparts on standard benchmarks. Our code is publicly available at http://github.com/microsoft/implicit_languagemodels .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。