用少量数据将大模型转为高效长文本混合架构，速度更快且更准。
Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
- 通过层优化实现从Transformer到混合RNN注意力的轻量知识蒸馏
- 仅用23亿词元就完成转换，长文本性能接近原模型
- 新位置编码与结构改进让模型在超长序列上表现更优
混合Transformer架构结合了softmax注意力与循环神经网络（RNN），在长序列建模中展现出良好的性能与吞吐平衡，但其广泛应用受限于大规模预训练的高昂成本。已有研究尝试通过参数迁移和知识蒸馏将预训练的softmax注意力模块转化为RNN模块，但这类方法需超过100亿词元的训练数据，且转换后的模型在长上下文场景下表现不佳——而这正是混合模型具备显著推理加速优势的领域。本文提出HALO（Hybrid Attention via Layer Optimization），一种将Transformer模型蒸馏为RNN-注意力混合模型的流程。进一步提出HypeNet，一种基于新型位置编码（HyPE）和多种架构优化的混合模型，在长序列泛化能力上表现卓越。使用HALO将Qwen3系列转换为HypeNet，仅需23亿词元（不足其预训练数据的0.01%），即可实现与原始模型相当的性能，同时在长上下文任务中兼具更高效率与更强表现。
原文摘要 · Abstract (English)
Hybrid Transformer architectures, which combine softmax attention blocks and recurrent neural networks (RNNs), have shown a desirable performance-throughput tradeoff for long-context modeling, but their adoption and studies are hindered by the prohibitive cost of large-scale pre-training from scratch. Some recent studies have shown that pre-trained softmax attention blocks can be converted into RNN blocks through parameter transfer and knowledge distillation. However, these transfer methods require substantial amounts of training data (more than 10B tokens), and the resulting hybrid models also exhibit poor long-context performance, which is the scenario where hybrid models enjoy significant inference speedups over Transformer-based models. In this paper, we present HALO (Hybrid Attention via Layer Optimization), a pipeline for distilling Transformer models into RNN-attention hybrid models. We then present HypeNet, a hybrid architecture with superior length generalization enabled by a novel position encoding scheme (named HyPE) and various architectural modifications. We convert the Qwen3 series into HypeNet using HALO, achieving performance comparable to the original Transformer models while enjoying superior long-context performance and efficiency. The conversion requires just 2.3B tokens, less than 0.01% of their pre-training data
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。