用RWKV-7替代Transformer，大幅降低时序建模的参数和训练成本。
BlackGoose Rimer: Harnessing RWKV-7 as a Simple yet Superior Replacement for Transformers in Large-Scale Time Series Modeling
- 将RWKV-7的时间混合与通道混合机制融入Timer模型
- 性能提升1.13至43.3倍，训练时间减少4.5倍，参数量仅为1/23
- 适合追求高效高精度时序建模的研究者与工业应用
时序模型在处理大规模复杂数据时面临显著挑战，类似大语言模型的扩展需求。尽管已有研究尝试Transformer、LSTM、GRU等架构，本文提出使用引入元学习机制的RWKV-7。通过将RWKV-7的时间混合与通道混合组件集成到基于Transformer的时序模型Timer中，实现约1.13至43.3倍的性能提升，训练时间减少4.5倍，且仅需1/23的参数量。代码与模型权重已开源，供后续研究使用。
原文摘要 · Abstract (English)
Time series models face significant challenges in scaling to handle large and complex datasets, akin to the scaling achieved by large language models (LLMs). The unique characteristics of time series data and the computational demands of model scaling necessitate innovative approaches. While researchers have explored various architectures such as Transformers, LSTMs, and GRUs to address these challenges, we propose a novel solution using RWKV-7, which incorporates meta-learning into its state update mechanism. By integrating RWKV-7's time mix and channel mix components into the transformer-based time series model Timer, we achieve a substantial performance improvement of approximately 1.13 to 43.3x and a 4.5x reduction in training time with 1/23 parameters, all while utilizing fewer parameters. Our code and model weights are publicly available for further research and development at https://github.com/Alic-Li/BlackGoose_Rimer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。