初始偏差让大模型在训练初期偏爱少数类别，影响学习方向。
Effects of Initialization Biases on Deep Neural Network Training Dynamics
- 发现未训练大模型存在初始偏好，仅少数类获得高概率输出。
- 两种抗标签错误损失函数在初始偏差下会失效，无法引导训练方向。
- 提示需关注初始化偏差与训练组件的相互作用，尤其早期阶段。
未经训练的大规模神经网络在随机初始化后，倾向于偏好少数类别，对这些类别分配高预测概率，而对其他类别则近似为零概率。这种现象被称为初始猜测偏差（Initial Guessing Bias），会影响模型在拟合数据粗粒度结构时的早期训练动态。训练所采用的损失函数对早期动态有显著影响。近期设计的两种鲁棒性损失函数——模糊损失（Blurry loss）和分段零损失（Piecewise-zero loss），虽旨在抵抗标签错误，但在面对初始偏差时可能丧失引导训练方向的能力。结果表明，损失函数的选择对网络早期训练具有决定性影响，凸显了在训练方案中需审慎考虑初始猜测偏差与其他组件的交互作用。
原文摘要 · Abstract (English)
Untrained large neural networks, just after random initialization, tend to favour a small subset of classes, assigning high predicted probabilities to these few classes and approximately zero probability to all others. This bias, termed Initial Guessing Bias, affects the early training dynamics, when the model is fitting to the coarse structure of the data. The choice of loss function against which to train the model has a large impact on how these early dynamics play out. Two recent loss functions, Blurry and Piecewise-zero loss, were designed for robustness to label errors but can become unable to steer the direction of training when exposed to this initial bias. Results indicate that the choice of loss function has a dramatic effect on the early phase training of networks, and highlights the need for careful consideration of how Initial Guessing Bias may interact with various components of the training scheme.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。