提出可解释的分层训练策略，让模型高效持续预训练。
Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training
- 通过分析各层表征演化与敏感度，定位任务执行位置。
- 浅层训练、深层冻结在双基准上优于全参数微调。
- 适合资源有限团队做模型持续训练与模块化设计。
选择性地对大语言模型进行分层更新是实现低成本持续预训练的关键，但确定哪些层应冻结或训练仍缺乏可解释的指导，属于经验性黑箱问题。为此，我们提出 LayerTracer——一个与架构无关的诊断框架，能够揭示各层表征的演化模式与稳定性，通过定位任务执行位置并量化层敏感度。分析发现，深层是任务执行的核心区域，且对破坏性更新具有高度稳定性。基于此，我们进行了三项受控的持续预训练实验，对比不同冻结-训练策略，结果表明：在 C-Eval 与 CMMLU 基准上，仅训练浅层而冻结深层的方法始终优于全参数微调及相反分配策略。此外，还开展混合模型案例研究，验证将高质量预训练模块置于深层能有效保留模型原有知识。本工作为资源受限团队提供了低成本、可解释的持续预训练分层参数分配方案，适用于模型持续训练与混合模型构建。
原文摘要 · Abstract (English)
Selective layer-wise updates are essential for low-cost continued pre-training of Large Language Models (LLMs), yet determining which layers to freeze or train remains an empirical black-box problem due to the lack of interpretable guidance. To address this issue, we propose LayerTracer, an architecture-agnostic diagnostic framework that reveals the evolution patterns of layer-wise representations and stability by locating task execution positions and quantifying layer sensitivity. Analysis results reveal that deep layers act as critical regions for task execution and maintain high stability against disruptive updates. Guided by this finding, we conduct three controlled continued pre-training trials to compare diverse freeze-train strategies, demonstrating that training shallow layers while freezing deep layers consistently outperforms full-parameter fine-tuning and the opposite allocation on both C-Eval and CMMLU benchmarks. We further present a hybrid model case study, which validates that placing high-quality pre-trained modules in deep layers effectively preserves inherent knowledge of the model. This work delivers a low-cost and interpretable solution for resource-constrained teams, offering actionable guidance for layer-wise parameter allocation in continued pre-training and hybrid model construction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。