通过动态生成合成数据提升小模型性能，训练更高效。
Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method
- 根据学生模型低效区域生成针对性合成数据。
- 6600万参数小模型在QNLI和SST-2上分别达91.2%和92.3%准确率。
- 跳过教师输入层，直接对向量表示进行压缩，提速降耗。
模型蒸馏可将大模型知识迁移到轻量级学生模型中，便于在资源受限环境部署。然而传统蒸馏方法常伴随计算开销大、泛化能力差的问题。本文提出一种自适应蒸馏框架，通过UMAP降维与近邻采样，识别学生模型嵌入空间中表现不佳的区域，并生成针对性合成样本以引导学习。为提升效率，设计轻量级师生接口，跳过教师输入层，直接对向量表示进行蒸馏。在标准NLP基准测试中，6600万参数的学生模型持续达到或超越基线，于QNLI上达91.2%，于SST-2上达92.3%，且训练所需轮次更少。结果表明，基于损失感知的数据增强与向量化蒸馏在高效模型压缩中具有显著潜力。
原文摘要 · Abstract (English)
Model distillation enables the transfer of knowledge from large-scale models to compact student models, facilitating deployment in resource-constrained environments. However, conventional distillation approaches often suffer from computational overhead and limited generalization. We propose a novel adaptive distillation framework that dynamically augments training data in regions of high student model loss. Using UMAP-based dimensionality reduction and nearest neighbor sampling, our method identifies underperforming regions in the embedding space and generates targeted synthetic examples to guide student learning. To further improve efficiency, we introduce a lightweight teacher-student interface that bypasses the teacher's input layer, enabling direct distillation on vectorized representations. Experiments across standard NLP benchmarks demonstrate that our 66M-parameter student model consistently matches or surpasses established baselines, achieving 91.2% on QNLI and 92.3% on SST-2, while training with fewer epochs. These results highlight the promise of loss-aware data augmentation and vectorized distillation for efficient and effective model compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。