arXiv:2504.08247cs.LGcs.CL2025-04被引 1

让RWKV-7支持可扩展的参数交互,实现线性复杂度下的持续增长。

Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner

  • 用自状态编码器将输入与参数动态融合,不新增可训练矩阵。
  • 通过扩展状态和参数令牌实现模型扩容,无需重新训练。
  • 适合需要高效、持续扩展的序列建模任务,如长文本生成。

基于状态的序列模型如RWKV-7提供了替代Transformer的方案,在保持线性复杂度的同时,在短上下文场景中展现出更强的表达能力,并能突破 ext{TC}^0复杂度类的限制。然而,其缺乏令牌-参数交互机制和原生可扩展性,限制了适应性和增长能力。本文提出 extbf{Meta-State},作为对RWKV-7的改进:以全状态驱动方式取代注意力机制,通过 extbf{自状态编码器}(SSE)将部分权重状态重用于编码令牌-参数交互,实现线性、状态驱动的融合,不引入新可训练矩阵或softmax操作,同时保留自回归特性。该方法支持通过扩展WKV状态和参数令牌实现渐进式模型缩放,复用已有参数无需再训练。本工作弥合了状态建模、令牌-参数交互与可扩展架构之间的差距,为具有线性复杂度和恒定内存消耗的高效、可适应序列建模提供灵活框架。

原文摘要 · Abstract (English)

State-based sequence models like RWKV-7 offer a compelling alternative to Transformer architectures, achieving linear complexity while demonstrating greater expressive power in short-context scenarios and enabling state tracking beyond the \(\text{TC}^0\) complexity class. However, RWKV-7 lacks mechanisms for token-parameter interactions and native scalability, limiting its adaptability and growth without retraining. In this paper, we propose \textbf{Meta-State}, a novel extension to RWKV-7 that replaces attention mechanisms with a fully state-driven approach, integrating token-parameter interactions through a \textbf{Self-State Encoder} (SSE) mechanism. The SSE repurposes a portion of the RWKV-7 Weighted Key-Value (WKV) state as transformation weights to encode token-parameter interactions in a linear, state-driven manner without introducing new trainable matrices or softmax operations, while preserving the autoregressive property of token processing. Meta-State supports progressive model scaling by expanding the WKV state and parameter tokens, reusing existing parameters without retraining. Our approach bridges the gap between state-based modeling, token-parameter interactions, and scalable architectures, offering a flexible framework for efficient and adaptable sequence modeling with linear complexity and constant memory usage.

RWKVMoE状态模型可扩展

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。