探索解码器型大模型中间层蒸馏的潜力与局限
A Study on Hidden Layer Distillation for Large Language Model Pre-Training
- 用教师模型中间层特征辅助学生模型预训练
- 中间层蒸馏在困惑度上优于传统输出蒸馏
- 适合关注隐层语义信息提取的研究者
知识蒸馏（KD）是训练大语言模型的关键技术，但多数研究仅依赖输出概率分布，忽视教师模型中间层的语义信息。虽然隐藏层蒸馏（HLD）在编码器结构中展现出潜力，但在大规模解码器架构预训练中的应用仍不明确。我们以Gemma3 3.4B为教师模型，使用123M和735M参数的学生模型，在最多168B tokens的C4数据集上进行计算可控实验，对比HLD、基于输出的KD及自监督基线。结果显示，HLD在下游任务上并未持续优于标准KD；然而，在所有共享超参配置下，其均带来系统性困惑度降低，表明可提取潜在表征信号，但要实现显著突破仍需新方法。
原文摘要 · Abstract (English)
Knowledge Distillation (KD) is a critical tool for training Large Language Models (LLMs), yet the majority of research focuses on approaches that rely solely on output logits, neglecting semantic information in the teacher's intermediate representations. While Hidden Layer Distillation (HLD) showed potential for encoder architectures, its application to decoder-only pre-training at scale remains largely unexplored. Through compute-controlled experiments, we benchmark HLD against logit-based KD and self-supervised baselines with Gemma3 3.4B as teacher and 123M and 735M students trained on up to 168B tokens from the C4 dataset. Our experiments show that HLD does not consistently outperform standard KD on downstream evaluation tasks. Nevertheless, we show that HLD can yield a systematic perplexity gain over KD across all shared-hyperparameter configurations, suggesting that a latent signal can be extracted, but a breakthrough may be needed for it to play a more significant role in LLM pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。