小模型加冻结大模型层，高效提升性能
Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models
- 将大模型的冻结层插入小模型，仅微调新部分
- 在多项任务上显著提升小模型表现，计算开销小
- 适合资源有限但需强泛化能力的应用场景
大语言模型（LLMs）具备丰富的语言知识和强泛化能力，但计算开销大，不适用于资源受限环境。小语言模型（SLMs）虽计算高效，但泛化能力不足。为此，我们提出PiFi框架：将一个冻结的LLM层集成到SLM中，并对融合模型进行特定任务微调，实现高性能的同时保持低计算成本。实验表明，PiFi在自然语言理解与生成任务中均带来稳定提升。结果还显示，该方法能有效利用LLM知识，增强对未见领域的泛化能力，促进语言能力迁移。
原文摘要 · Abstract (English)
Large language models (LLMs) are renowned for their extensive linguistic knowledge and strong generalization capabilities, but their high computational demands make them unsuitable for resource-constrained environments. In contrast, small language models (SLMs) are computationally efficient but often lack the broad generalization capacity of LLMs. To bridge this gap, we propose PiFi, a novel framework that combines the strengths of both LLMs and SLMs to achieve high performance while maintaining efficiency. PiFi integrates a single frozen layer from an LLM into a SLM and fine-tunes the combined model for specific tasks, boosting performance without a significant increase in computational cost. We show that PiFi delivers consistent performance improvements across a range of natural language processing tasks, including both natural language understanding and generation. Moreover, our findings demonstrate PiFi's ability to effectively leverage LLM knowledge, enhancing generalization to unseen domains and facilitating the transfer of linguistic abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。