提出新型循环单元，让模型能准确处理任意长度序列。
Length Generalization with Log-Depth Recurrent Units
- 用并行化归约机制替代传统递归，避免位置偏差
- 21个任务中18个实现100%外推精度，3个超99.9%
- 适合需要长序列泛化的建模场景
长度泛化仍是神经网络的难题：循环模型易受位置偏差影响，而变换器受限于固定计算深度。正则语言常被用作评估长度泛化的测试基准，因标签预测可验证任意序列长度。本文提出MLP-LDRU，一种对数深度循环单元，通过并行归约近似递归，捕捉一类关联性偏倚算子。我们在21个正则语言任务（含标准基准与新提出的前缀语言）上评估MLP-LDRU，当最大训练长度增加时，在18个任务上达到100%分布外准确率，其余3个任务至少99.9%，显著优于可比的循环与注意力模型。此外，在ListOps及NLP分类基准上，该模型也表现良好。
原文摘要 · Abstract (English)
Length generalization remains a persistent challenge for neural networks: recurrent models tend to suffer from positional biases, while transformers are constrained by fixed computational depth. Regular languages provide a frequently used testbed for evaluating length generalization, as label prediction can be checked for any sequence length. We propose MLP-LDRU, a type of Log-Depth Recurrent Unit, which captures a class of associativity-biased operators designed to approximate recurrence through parallel reduction. We evaluate MLP-LDRU on 21 regular-language tasks, consisting of standard benchmarks and new prefix languages, where it achieves 100% out-of-distribution accuracy on 18 tasks and at least 99.9% on the remaining 3 when increasing max training length, outperforming comparable recurrent and attention-based models. We further evaluate MLP-LDRU beyond regular languages on ListOps and NLP classification benchmarks, where it performs competitively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。