arXiv:2605.06384cs.LGcs.AI2026-05

提出新型循环网络,解决长序列建模中表达力与稳定性难题。

MinMax Recurrent Neural Cascades

  • 基于MinMax代数构建新型递归结构,兼具强表达力与高效计算。
  • 可完美求解合成任务,支持长序列泛化,梯度不消失且状态稳定。
  • 适合需要长期依赖与高稳定性的实际序列建模任务。

我们提出MinMax递归神经级联(MinMax RNCs),一种基于MinMax代数的新式递归神经网络。其形式表达能力对应正则语言,是有限记忆系统的最大表达力;在保持递归计算的同时,支持对数深度、线性工作量的并行扫描计算;状态和激活值对任意序列长度均有界;损失梯度几乎处处存在且有界;状态梯度不会消失——过去状态对当前状态的梯度范数可恒为1,与时间距离无关。实验证明这些理论特性带来优异表现:能完美解决合成任务,泛化至长序列,并优于基线模型。我们还训练了一个1.12亿参数的MinMax RNC用于下一个词预测,性能与同类规模模型相当,初步证明该方法可扩展至真实序列建模任务。

原文摘要 · Abstract (English)

We introduce MinMax Recurrent Neural Cascades (MinMax RNCs), a class of recurrent neural networks built from a novel form of recurrence over the MinMax algebra. We show that MinMax RNCs enjoy key properties that are difficult to obtain simultaneously: strong formal expressivity, efficient evaluation, stable dynamics, and non-vanishing state gradients. First, their formal expressivity corresponds to the regular languages, arguably the maximal expressivity for finite-memory systems. Second, in addition to evaluation in recurrent form, they also admit parallel-scan evaluation with logarithmic depth and linear work in the input length. Third, their states and activations are uniformly bounded for all sequence lengths. Fourth, their loss gradients exist almost everywhere and are uniformly bounded for all sequence lengths. Fifth, they do not exhibit vanishing state gradients: the gradient of a state with respect to a past state can retain norm one independently of the temporal distance between the states. Empirically, we find that these theoretical properties translate into strong practical performance. MinMax RNCs solve the considered synthetic tasks perfectly, generalise to long sequences, and outperform the recurrent baselines considered in our experiments. We also train a 112M-parameter MinMax RNC for next-token prediction, obtaining competitive performance for its size and providing initial evidence that MinMax recurrence can scale to real-world sequence-modelling tasks.

递归网络序列建模梯度稳定形式表达

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。