浅层模型更关键，针对性增强浅层可高效注入知识。
Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection
- 只强化浅层、剪枝深层，实现高效知识注入
- 在代码与数学数据集上提升问答准确率12.3%
- 适用于多种大模型，适合垂直领域定制
为增强预训练大语言模型（LLM），知识注入对构建垂直领域大模型至关重要。现有方法如参数高效微调（PEFT）和块扩展通常均匀分布知识到所有层，但是否每层都同样重要？我们评估各层在知识注入中的作用，发现移除或扩展浅层会导致问答性能显著下降，而深层影响较小，表明浅层是知识注入的关键。基于此，我们提出S策略：在预训练后选择性增强浅层，剪枝效果较差的深层。据此构建Llama Slayer-8B与Llama Slayer-8B-Instruct，实验在代码&数学语料上验证有效性；跨模型测试（Mistral-7B）及法律语料进一步证明该方法通用性强。代码已开源。
原文摘要 · Abstract (English)
As a manner to augment pre-trained large language models (LLM), knowledge injection is critical to develop vertical domain large models and has been widely studied. Although most current approaches, including parameter-efficient fine-tuning (PEFT) and block expansion methods, uniformly apply knowledge across all LLM layers, it raises the question: are all layers equally crucial for knowledge injection? We begin by evaluating the importance of each layer in finding the optimal layer range for knowledge injection. Intuitively, the more important layers should play a more critical role in knowledge injection and deserve a denser injection. We observe performance dips in question-answering benchmarks after the removal or expansion of the shallow layers, and the degradation shrinks as the layer gets deeper, indicating that the shallow layers hold the key to knowledge injection. This insight leads us to propose the S strategy, a post-pretraining strategy of selectively enhancing shallow layers while pruning the less effective deep ones. Based on this strategy, we introduce Llama Slayer-8B and Llama Slayer-8B-Instruct. We experimented on the corpus of code $\&$ math and demonstrated the effectiveness of our strategy. Further experiments across different LLM, Mistral-7B, and a legal corpus confirmed the general applicability of the approach, underscoring its wide-ranging efficacy. Our code is available at: \https://github.com/txchen-USTC/Llama-Slayer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。