用递归结构模拟注意力，小模型实现高效高精度
Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structures
- 基于xLSTM的递归结构逼近Transformer注意力参数
- 蒸馏训练后性能接近大模型，计算开销更低
- 适合资源受限场景下的轻量级语言模型应用
当前自然语言处理以Transformer模型为主导，但基于循环机制的新架构（如xLSTM和Mamba）作为注意力模型的替代方案逐渐兴起。尽管计算方式不同于注意力机制，这些循环模型仍能取得良好效果，甚至在某些情况下超越最先进的注意力模型。本文提出Distil-xLSTM，一种基于xLSTM的小型语言模型（SLM），通过从大型语言模型（LLM）中蒸馏知识进行训练，展现出令人瞩目的性能，同时具备计算和规模效率优势。Distil-xLSTM专注于利用其递归序列混合组件近似Transformer模型的注意力参数化，仅需少量训练即可获得优异结果。
原文摘要 · Abstract (English)
The current era of Natural Language Processing (NLP) is dominated by Transformer models. However, novel architectures relying on recurrent mechanisms, such as xLSTM and Mamba, have been proposed as alternatives to attention-based models. Although computation is done differently than with the attention mechanism mechanism, these recurrent models yield good results and sometimes even outperform state-of-the-art attention-based models. In this work, we propose Distil-xLSTM, an xLSTM-based Small Language Model (SLM) trained by distilling knowledge from a Large Language Model (LLM) that shows promising results while being compute and scale efficient. Our Distil-xLSTM focuses on approximating a transformer-based model attention parametrization using its recurrent sequence mixing components and shows good results with minimal training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。