对比两种模型压缩方法，发现无标签蒸馏效果不输有标签微调。
Constrained Edge AI Deployment: Fine-Tuning vs Distillation for LLM Compression
- 仅对MLP层做层间L2正则剪枝，比较交叉熵微调与基于KL散度的自蒸馏。
- 在相同剪枝条件下，自蒸馏在CommonsenseQA上准确率媲美甚至超过有标签微调。
- 适合无网络连接或数据受限的边缘设备部署场景。
当前基础模型常通过结构化剪枝与重训练结合实现压缩，以满足边缘设备严格的计算、内存和连通性限制。尽管最先进的剪枝方案作用于整个Transformer结构，本文仅对MLP模块采用简单的逐层L2范数剪枝作为基准。研究重点并非追求最大压缩比，而是隔离重训练损失函数的影响：(i) 基于交叉熵（L2PFT）的微调需标注数据；(ii) 基于KL散度的自蒸馏仅依赖教师模型输出（无需标签）（L2PSD）。在适用于间歇或断网环境的OLMo2-7B-SFT模型上评估两者，结果表明，在相同剪枝策略下，基于KL的蒸馏在测试准确率上达到或超越交叉熵微调，证明即使采用基础的MLP剪枝，损失函数的选择仍显著影响资源受限环境下压缩模型的恢复性能。
原文摘要 · Abstract (English)
Modern foundational models are often compressed via a combination of structured pruning and re-training to meet the strict compute, memory, and connectivity constraints of edge deployments. While state-of-the-art pruning schemes target the entire Transformer, we adopt a simple, layer-wise L2-norm pruning on only the MLP blocks as a fixed baseline. Our focus is not on achieving maximal compression, but on isolating the impact of the re-training loss function: (i) Fine-tuning with Cross- Entropy (L2PFT), which requires labeled data, versus (ii) Self-Distillation with KL-divergence, which leverages only teacher logits (no labels) (L2PSD). We evaluate both pipelines on the OLMo2- 7B-SFT model for CommonsenseQA suitable for intermittent or denied connectivity scenarios typical of edge networks. Under identical pruning schedules, KL-based distillation matches or exceeds CE fine-tuning in test accuracy, demonstrating that, even with a basic MLP-only pruning, the choice of loss function materially affects compressed model recovery in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。