用RNN+注意力机制重构Transformer，实现更高效的语言建模。
ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer
- 基于RWKV-7架构的混合RNN与注意力模型，提升表达能力。
- 32B参数模型仅用8小时训练完成，16张AMD MI300X显卡即可达成。
- 可将大模型知识迁移至小模型，适合追求高效部署的研究者。
现有混合二次与亚二次注意力模型在多头架构中已超越Transformer与线性RNN，主要聚焦于降低键值对（KV）复杂度并提升效率。为进一步探索表达能力，我们基于Qwen 2.5蒸馏出一系列纯原生RWKV-7注意力模型，使RNN具备更强表达力，并展现超越Transformer的状态追踪能力。以基于RWKV-6架构的QRWK 32B为例，仅用16张AMD MI300X GPU，在8小时内完成全部知识处理，性能保持与Qwen 2.5相当。该蒸馏过程适用于任意LLM，可将大模型知识高效迁移到小模型，且所需令牌数更少。本文详述构建流程与见解，推动更强基础模型发展。当前工作仍在持续更新中，模型权重与源码已公开于GitHub及Hugging Face。
原文摘要 · Abstract (English)
As is known, hybrid quadratic and subquadratic attention models in multi-head architectures have surpassed both Transformer and Linear RNN models , with these works primarily focusing on reducing KV complexity and improving efficiency. For further research on expressiveness, we introduce our series of models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which aims to make RNN more expressive and demonstrates state tracking ability beyond transformers. We work with QRWK 32B based on RWKV-6 architecture, another approach that reduces the entire knowledge processing time to just 8 hours using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the distillation process can utilize any LLM, not just Qwen, and enables knowledge transfer from larger LLMs to smaller ones with more fewer tokens. We will explain the detailed process and share our insights on building more powerful foundation models. Please note that this is an ongoing work that will be updated continuously. The model checkpoints and source code are available at \href{https://github.com/yynil/RWKVInside}{https://github.com/yynil/RWKVInside}, \href{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。