找出大模型对齐中最重要的层，提升微调效率。
Understanding Layer Significance in LLM Alignment
- 用二值掩码识别微调时各层的重要性。
- 不同数据集下重要层重叠率达90%以上。
- 只调关键层可大幅提效，性能损失小。
通过监督微调对大语言模型（LLMs）进行对齐是使其适应特定应用的关键。近期研究表明，对齐主要改变模型的表达风格而非基础知识，仅影响模型的某些组件。为深入理解对齐如何在细粒度层面影响模型行为，我们提出ILa方法，通过学习每层参数变化的二值掩码来识别对齐过程中各层的重要性。实验表明，尽管对齐数据集差异显著，由ILa识别的重要层之间重合度接近90%，揭示了大模型对齐中的根本模式。结果还显示，冻结非关键层能提升整体性能，而仅微调关键层即可显著提高微调效率且性能损失极小。最后，我们讨论这些发现从模型对齐延伸至推理能力的潜力。
原文摘要 · Abstract (English)
Aligning large language models (LLMs) through supervised fine-tuning is essential for tailoring them to specific applications. Recent studies suggest that alignment primarily adjusts a model's presentation style rather than its foundational knowledge, indicating that only certain components of the model are significantly impacted. To uncover how alignment affects model behavior at a granular level, we propose identifying which layers within LLMs are most critical to the alignment process. Our approach, named ILA, involves learning a binary mask for the parameter changes in each layer during alignment, as an indicator of layer significance. Experimental results reveal that, despite substantial differences in alignment datasets, the important layers of a model identified by ILA exhibit nearly 90\% overlap, highlighting fundamental patterns in LLM alignment. The results also indicate that freezing non-essential layers improves overall model performance, while selectively tuning the most critical layers significantly enhances fine-tuning efficiency with minimal performance loss. Finally, we discuss how these findings extend from LLM alignment to reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。