提出轻量级基准测试，揭示线性循环模型的选通机制如何影响性能。
Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalization
- 设计可定制合成任务，评估模型对关键信息的选择能力。
- 发现门控与快速遗忘提升记忆，状态内混合对泛化至关重要。
- 适合研究高效序列模型的架构设计与行为机制。
线性循环神经网络因其可并行训练及推理时恒定的内存与计算开销，成为Transformer中softmax注意力机制的高效替代方案。尽管迭代改进引入了越来越多的结构机制，导致复杂度与计算成本上升，但这些模型之间的系统性对比仍有限。现有基准任务或过于简单，无法揭示显著差异，或资源消耗过大难以实验。本文提出线性循环模型的细化分类，并引入SelectivBench——一组轻量级、可定制的合成基准任务，用于系统评估小到中等规模序列模型的选择性。该基准通过基于规则的语法生成具有可调复杂度的序列，包含故意违反转移规则的不规则间隔。在线性循环模型上的评估结果与大规模语言任务表现一致。分析表明：门控与快速遗忘机制促进回忆，状态内通道混合对选择性非必需但对泛化至关重要，而softmax注意力因序列长度相关记忆容量仍占主导。该基准支持针对线性循环模型的高效靶向探索，并提供可控环境研究大规模评估中的行为模式。代码已开源。
原文摘要 · Abstract (English)
Linear recurrent neural networks have emerged as efficient alternatives to the original Transformer's softmax attention mechanism, thanks to their highly parallelizable training and constant memory and computation requirements at inference. Iterative refinements of these models have introduced an increasing number of architectural mechanisms, leading to increased complexity and computational costs. Nevertheless, systematic direct comparisons among these models remain limited. Existing benchmark tasks are either too simplistic to reveal substantial differences or excessively resource-intensive for experimentation. In this work, we propose a refined taxonomy of linear recurrent models and introduce SelectivBench, a set of lightweight and customizable synthetic benchmark tasks for systematically evaluating sequence models. SelectivBench specifically evaluates selectivity in sequence models at small to medium scale, such as the capacity to focus on relevant inputs while ignoring context-based distractors. It employs rule-based grammars to generate sequences with adjustable complexity, incorporating irregular gaps that intentionally violate transition rules. Evaluations of linear recurrent models on SelectivBench reveal performance patterns consistent with results from large-scale language tasks. Our analysis clarifies the roles of essential architectural features: gating and rapid forgetting mechanisms facilitate recall, in-state channel mixing is unnecessary for selectivity, but critical for generalization, and softmax attention remains dominant due to its memory capacity scaling with sequence length. Our benchmark enables targeted, efficient exploration of linear recurrent models and provides a controlled setting for studying behaviors observed in large-scale evaluations. Code is available at https://github.com/symseqbench/selectivbench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。