让大模型推理过程对齐，提升小模型压缩效果
DistillLens: Symmetric Knowledge Distillation Through Logit Lens
- 用日志透镜将中间层映射到词表空间，实现师生模型推理路径对齐
- 对称损失约束使模型既不盲目自信也不过度犹豫，保留关键信息通道
- 在GPT-2和Llama上均超越传统方法，适合模型压缩与高效推理场景
标准知识蒸馏通过优化最终输出压缩大型语言模型，但通常将教师模型的中间层思维过程视为黑箱。尽管基于特征的蒸馏试图弥合这一差距，现有方法（如MSE和非对称KL散度）忽略了最终输出所需的丰富不确定性分布。本文提出DistillLens框架，通过对称方式对齐学生与教师模型的演化推理过程。通过日志透镜将中间隐藏状态投影至词汇空间，并使用对称散度目标强制结构对齐。分析证明该约束施加双向惩罚，防止过自信与欠自信，同时保留高熵信息通路以支持最终推断。在GPT-2和Llama架构上的大量实验表明，DistillLens在多种指令跟随基准上持续优于标准知识蒸馏和特征迁移基线。代码已开源。
原文摘要 · Abstract (English)
Standard Knowledge Distillation (KD) compresses Large Language Models (LLMs) by optimizing final outputs, yet it typically treats the teacher's intermediate layer's thought process as a black box. While feature-based distillation attempts to bridge this gap, existing methods (e.g., MSE and asymmetric KL divergence) ignore the rich uncertainty profiles required for the final output. In this paper, we introduce DistillLens, a framework that symmetrically aligns the evolving thought processes of student and teacher models. By projecting intermediate hidden states into the vocabulary space via the Logit Lens, we enforce structural alignment using a symmetric divergence objective. Our analysis proves that this constraint imposes a dual-sided penalty, preventing both overconfidence and underconfidence while preserving the high-entropy information conduits essential for final deduction. Extensive experiments on GPT-2 and Llama architectures demonstrate that DistillLens consistently outperforms standard KD and feature-transfer baselines on diverse instruction-following benchmarks. The code is available at https://github.com/manishdhakal/DistillLens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。