软标签训练比硬标签少用大量神经元，尤其在难分类数据上优势明显。
A Theoretical Analysis of Soft-Label vs Hard-Label Training in Neural Networks
- 用软标签替代硬标签，能显著降低模型对神经元数量的需求。
- 理论证明：软标签只需 $O(1/γ^2ε)$ 个神经元，硬标签需 $O(1/γ^4 ext{ln}(1/ε))$。
- 适合研究知识蒸馏、模型压缩及小样本学习的科研人员参考。
知识蒸馏通过让小型学生模型从预训练的大教师模型中学习，已取得显著实证成功。然而，一个关键问题仍未解决：为何使用教师模型输出的软标签进行训练，所需神经元远少于直接使用硬标签训练小型网络？本文首先在二分类任务中使用简单神经网络进行实验，结果表明软标签训练在准确率上始终优于硬标签，且数据越难分类，差距越大。随后，基于两层神经网络模型，提出理论分析：软标签训练在梯度下降下仅需 $O(1/(γ^2 ε))$ 个神经元即可使平均分类损失低于任意 $ε>0$；而硬标签训练则需 $O(1/γ^4 ext{ln}(1/ε))$ 个神经元，该结论基于 extcite{ji2020polylogarithmic} 的梯度下降分析改进而来。当 $γ ≤ ε$（即数据难以分类）时，软标签训练所需的神经元数量可大幅减少。最后，深度网络上的实验进一步验证了这些理论发现。
原文摘要 · Abstract (English)
Knowledge distillation, where a small student model learns from a pre-trained large teacher model, has achieved substantial empirical success since the seminal work of \citep{hinton2015distilling}. Despite prior theoretical studies exploring the benefits of knowledge distillation, an important question remains unanswered: why does soft-label training from the teacher require significantly fewer neurons than directly training a small neural network with hard labels? To address this, we first present motivating experimental results using simple neural network models on a binary classification problem. These results demonstrate that soft-label training consistently outperforms hard-label training in accuracy, with the performance gap becoming more pronounced as the dataset becomes increasingly difficult to classify. We then substantiate these observations with a theoretical contribution based on two-layer neural network models. Specifically, we show that soft-label training using gradient descent requires only $O\left(\frac{1}{γ^2 ε}\right)$ neurons to achieve a classification loss averaged over epochs smaller than some $ε> 0$, where $γ$ is the separation margin of the limiting kernel. In contrast, hard-label training requires $O\left(\frac{1}{γ^4} \cdot \ln\left(\frac{1}ε\right)\right)$ neurons, as derived from an adapted version of the gradient descent analysis in \citep{ji2020polylogarithmic}. This implies that when $γ\leq ε$, i.e., when the dataset is challenging to classify, the neuron requirement for soft-label training can be significantly lower than that for hard-label training. Finally, we present experimental results on deep neural networks, further validating these theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。