将Transformer模型转换为线性复杂度模型并联合微调，保持原有性能。
Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity
- 通过层间蒸馏方法，同时完成模型转换与任务微调。
- 在语言和语音任务中，性能接近原始Transformer模型。
- 适合需要高效推理的语音与语言模型部署场景。
Linformer和Mamba等架构近年来作为Transformer的线性时间替代方案表现出竞争力。然而，这些架构的大规模预训练模型往往不可用，尤其是在非文本领域。为此，我们提出跨架构分层蒸馏（CALD）方法，能够同时将Transformer模型转换为线性时间替代模型，并对其进行目标任务微调。我们还比较了多种引导微调的策略，以最优保留原模型的推理能力。不同策略在目标模型使用方式和参数轨迹上存在差异。在一系列语言处理、语言建模和语音处理的实证研究中，我们表明CALD能有效恢复原模型性能，且引导策略对结果有显著影响。部分变异原因也被分析解释。
原文摘要 · Abstract (English)
Architectures such as Linformer and Mamba have recently emerged as competitive linear time replacements for transformers. However, corresponding large pretrained models are often unavailable, especially in non-text domains. To remedy this, we present a Cross-Architecture Layerwise Distillation (CALD) approach that jointly converts a transformer model to a linear time substitute and fine-tunes it to a target task. We also compare several means to guide the fine-tuning to optimally retain the desired inference capability from the original model. The methods differ in their use of the target model and the trajectory of the parameters. In a series of empirical studies on language processing, language modeling, and speech processing, we show that CALD can effectively recover the result of the original model, and that the guiding strategy contributes to the result. Some reasons for the variation are suggested.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。