arXiv:2605.17108cs.LG2026-05

用递归树结构让LSTM并行化,解决长序列处理效率问题。

Parallel Recursive LSTM

  • 用平衡树结构递归合并隐状态,替代传统顺序计算。
  • 在形式语言任务上超越RNN、LSTM和Transformer,且无二次开销。
  • 适合需要长序列建模又追求高效并行的场景。

Transformers 通过自注意力实现高效并行,但其时间与内存成本随序列长度呈二次增长,限制了长序列应用。传统循环模型如LSTM虽具强状态追踪能力,但严格顺序计算难以并行。本文提出并行递归LSTM(PR-LSTM),将左到右的递归替换为在平衡计算树上的递归非线性状态组合。输入标记先独立映射为隐状态,再通过可学习门控组合块递归合并。该结构采用并行扫描的归约模式作为固定执行调度,而非假设关联性递推。结果使递归并行深度从线性降至对数级,同时保持非线性门控状态表示。实验表明,PR-LSTM在形式语言基准上表现出强序列长度泛化能力,解决的任务数量超过标准RNN、LSTM及Transformer基线,且避免了注意力的二次复杂度。这表明,通过层次化重构,循环计算可在不约束转移动态为线性或关联形式的前提下实现并行。

原文摘要 · Abstract (English)

Transformers have become the dominant architecture for sequence modeling by using self-attention to enable expressive and highly parallel processing. However, the resulting quadratic time and memory costs limit efficiency in long-context settings. Recurrent models such as LSTMs provide explicit nonlinear state updates and strong state-tracking capabilities, yet their strictly sequential computation limits parallelism. We introduce the Parallel Recursive LSTM (PR-LSTM), a hierarchical recurrent architecture that replaces left-to-right recurrence with recursive nonlinear state composition over a balanced computation tree. Tokens are first mapped independently to latent states, which are then recursively merged by a learned gated composition block. This structure uses the reduction pattern underlying parallel scans as a fixed execution schedule, rather than assuming an associative recurrence. As a result, PR-LSTM retains nonlinear gated state representations while reducing recurrent parallel depth from linear to logarithmic. Empirically, PR-LSTM achieves strong sequence-length generalization on formal-language benchmarks, solving more tasks than standard RNN, LSTM, and Transformer baselines, while avoiding the quadratic scaling of attention. These results suggest that recurrent computation can be reorganized hierarchically to expose parallelism without restricting the transition dynamics to linear or associative forms.

序列建模LSTM并行计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。