提出新型闭环控制的双线性RNN，提升序列建模效率与性能。
Comba: Improving Bilinear RNNs with Closed-loop Control
- 基于闭环控制理论设计新结构，融合状态与输出反馈修正。
- 340M/1.3B参数模型在语言与视觉任务上均表现更优。
- 支持硬件高效并行，适合大规模序列建模应用。
近年来，如Gated DeltaNet、TTT和RWKV-7等高效序列建模方法通过引入Delta学习规则监督循环记忆管理，实现了性能提升。与先前的状态空间模型（如Mamba）和门控线性注意力（如GLA）不同，这些模型在循环状态与键向量间引入交互，结构上类似双线性系统。本文首次系统分析双线性RNN的优势与局限，并基于闭环控制理论提出新型双线性RNN变体Comba，采用标量加低秩状态转移，结合状态反馈与输出反馈修正。我们还在Triton中实现硬件高效的分块并行核，在大规模语料上训练了340M/1.3B参数模型。Comba在语言与视觉建模任务中均展现出更优性能与计算效率。
原文摘要 · Abstract (English)
Recent efficient sequence modeling methods such as Gated DeltaNet, TTT, and RWKV-7 have achieved performance improvements by supervising the recurrent memory management through Delta learning rule. Unlike previous state-space models (e.g., Mamba) and gated linear attentions (e.g., GLA), these models introduce interactions between the recurrent state and the key vector, structurally resembling bilinear systems. In this paper, we first introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models. Then, based on closed-loop control theory, we propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections. We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on large-scale corpus. Comba demonstrates superior performance and computation efficiency in both language and vision modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。