调整损失权重可改善小规模微调中的类别不平衡问题。
Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining
- 在最后层微调中，通过动态调整损失权重来缓解类别偏差。
- 当模型参数量适中时，合理加权能使各分类性能更均衡。
- 适用于数据稀疏、需精准调控的下游任务场景。
随着大规模机器学习模型在判别任务中能力增强，其对训练数据引入偏差的抵抗能力受到越来越多关注。先前研究表明，存在两种极端的参数化情况：人口（欠参数化）设置下损失加权最优，而可分的过参数化设置下损失加权无效。本文研究了最后层微调（LLR）这一介于两者之间的场景——此时未见的微调数据通常不可分，且模型规模适中。理论与实证表明，尽管该场景处于中间状态，损失加权依然有效，但权重必须考虑模型的相对过参数化程度。
原文摘要 · Abstract (English)
While machine learning models become more capable in discriminative tasks at scale, their ability to overcome biases introduced by training data has come under increasing scrutiny. Previous results suggest that there are two extremes of parameterization with very different behaviors: the population (underparameterized) setting where loss weighting is optimal and the separable overparameterized setting where loss weighting is ineffective at ensuring equal performance across classes. This work explores the regime of last layer retraining (LLR) in which the unseen limited (retraining) data is frequently inseparable and the model proportionately sized, falling between the two aforementioned extremes. We show, in theory and practice, that loss weighting is still effective in this regime, but that these weights \emph{must} take into account the relative overparameterization of the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。