用极少数据快速将Transformer转为线性注意力模型，性能接近原版。
RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale
- 仅需350-700M tokens即可蒸馏转换，不足原模型训练量的0.005%。
- 72B模型转换成本低于2000美元，推理质量与原Transformer接近。
- 支持7B/32B/72B多尺寸模型，已开源，适合高效部署场景。
我们提出RADLADS，一种快速将softmax注意力Transformer转换为线性注意力解码器的协议，包含两种新RWKV变体架构，并成功将流行的Qwen2.5系列7B、32B和72B模型转化为线性注意力版本。转换过程仅需350-700M token，不足原模型训练量的0.005%。72B模型转换成本低于2000美元，推理质量保持与原模型相近。在标准基准测试中，这些模型在同规模线性注意力模型中表现最优。所有模型已在HuggingFace上以Apache 2.0许可证发布（72B模型除外，受Qwen许可协议约束）。代码与模型地址：https://github.com/recursal/RADLADS-paper
原文摘要 · Abstract (English)
We present Rapid Attention Distillation to Linear Attention Decoders at Scale (RADLADS), a protocol for rapidly converting softmax attention transformers into linear attention decoder models, along with two new RWKV-variant architectures, and models converted from popular Qwen2.5 open source models in 7B, 32B, and 72B sizes. Our conversion process requires only 350-700M tokens, less than 0.005% of the token count used to train the original teacher models. Converting to our 72B linear attention model costs less than \$2,000 USD at today's prices, yet quality at inference remains close to the original transformer. These models achieve state-of-the-art downstream performance across a set of standard benchmarks for linear attention models of their size. We release all our models on HuggingFace under the Apache 2.0 license, with the exception of our 72B models which are also governed by the Qwen License Agreement. Models at https://huggingface.co/collections/recursal/radlads-6818ee69e99e729ba8a87102 Training Code at https://github.com/recursal/RADLADS-paper
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。