将大模型蒸馏为线性xLSTM架构，实现性能几乎无损甚至超越。
Effective Distillation to Hybrid xLSTM Architectures
- 通过合并线性化专家构建单一xLSTM学生模型，提升蒸馏效果。
- 在多个下游任务上，学生模型接近或超过教师模型性能。
- 适合追求高效低耗大模型替代方案的研究者与开发者。
已有大量研究尝试将二次复杂度的注意力型大语言模型（LLMs）蒸馏至次二次的线性化架构中。然而，尽管投入广泛，此类蒸馏模型在诸多下游任务上仍难以达到教师模型的性能水平。本文旨在实现无损蒸馏，以任务集上学生与教师的容错胜平率差值为定义标准。为此，我们提出一种针对xLSTM架构学生的有效蒸馏流程，引入额外的合并阶段，将各自独立线性化的专家整合为单一模型。通过在Llama、Qwen和Olmo系列的基础及指令微调模型上验证，我们的xLSTM学生模型在多数设置下恢复了教师模型的绝大部分性能，甚至在部分下游任务上实现超越。该工作为更节能、低成本的Transformer替代方案迈出关键一步。
原文摘要 · Abstract (English)
There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. To this end, we introduce an effective distillation pipeline for xLSTM-based students. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。