RWKV-X用线性复杂度实现百万级长文本生成，兼顾效率与性能。
RWKV-X: A Linear Complexity Hybrid Language Model
- 融合RWKV与稀疏注意力，训练线性、推理常数时间复杂度
- 64K上下文下通过64K passkey测试，准确率接近完美
- 适合需要超长序列处理的通用语言模型研究者
本文提出RWKV-X，一种新型混合架构，结合了RWKV在短程建模中的高效性与设计用于捕捉长程上下文的稀疏注意力机制。与以往依赖全注意力层并保持二次复杂度的混合方法不同，RWKV-X在训练中实现线性时间复杂度，在推理解码中实现常数时间复杂度。我们证明，当在64K-token序列上持续预训练时，RWKV-X在64K passkey检索基准上达到近完美的准确率。它在长上下文基准上持续优于先前的RWKV-7模型，同时在短上下文任务上仍保持强大性能。这些结果表明，RWKV-X是通用语言建模的可扩展且高效的骨干网络，能够以稳定的速度和内存使用解码长达100万标记的序列。为促进进一步研究与分析，我们已将检查点和相关代码公开于：https://github.com/howard-hou/RWKV-X。
原文摘要 · Abstract (English)
In this paper, we introduce RWKV-X, a novel hybrid architecture that combines the efficiency of RWKV for short-range modeling with a sparse attention mechanism designed to capture long-range context. Unlike previous hybrid approaches that rely on full attention layers and retain quadratic complexity, RWKV-X achieves linear-time complexity in training and constant-time complexity in inference decoding. We demonstrate that RWKV-X, when continually pretrained on 64K-token sequences, achieves near-perfect accuracy on the 64K passkey retrieval benchmark. It consistently outperforms prior RWKV-7 models on long-context benchmarks, while maintaining strong performance on short-context tasks. These results highlight RWKV-X as a scalable and efficient backbone for general-purpose language modeling, capable of decoding sequences up to 1 million tokens with stable speed and memory usage. To facilitate further research and analysis, we have made the checkpoints and the associated code publicly accessible at: https://github.com/howard-hou/RWKV-X.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。