RWKV-7用动态状态实现高效建模，29亿参数刷新多语言表现
RWKV-7 "Goose" with Expressive Dynamic State Evolution
- 引入向量门控与上下文学习率的新型增量规则
- 29亿参数模型在多语言任务上达到30亿级新标杆
- 支持正则语言识别，适合追求效率的部署场景
我们提出RWKV-7 'Goose'，一种具有恒定内存和每标记恒定推理时间的序列建模架构。尽管训练所用标记数远少于其他顶级模型，我们的29亿参数语言模型在多语言任务上达到了30亿参数级别的新基准，并在英语下游任务上匹配当前30亿参数的领先表现。RWKV-7引入了新的广义增量规则，包含向量值门控和上下文学习率，以及放松的值替换规则。我们证明，RWKV-7能够执行状态追踪并识别所有正则语言，同时保持训练的可并行性，这在标准复杂度假设下超越了Transformer的$\mathsf{TC}^0$能力限制。为展示其语言建模能力，我们还发布了扩展的开源3.1万亿标记多语言语料库，并在该数据集上训练了从1.9亿到29亿参数的四个RWKV-7模型。为促进开放性、可复现性和采用，我们已将模型及数据组件发布至https://huggingface.co/RWKV，训练与推理代码发布于https://github.com/RWKV/RWKV-LM，均采用Apache 2.0许可。
原文摘要 · Abstract (English)
We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens than other top models, our 2.9 billion parameter language model achieves a new 3B SoTA on multilingual tasks and matches the current 3B SoTA on English language downstream performance. RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited to $\mathsf{TC}^0$. To demonstrate RWKV-7's language modeling capability, we also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on this dataset. To foster openness, reproduction, and adoption, we release our models and dataset component listing at https://huggingface.co/RWKV, and our training and inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0 License.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。