最后一层微调能提升少数群体表现，关键在于验证集的组别平衡。
On the Unreasonable Effectiveness of Last-layer Retraining
- 只重训网络最后一层，用验证集数据调整模型决策。
- 即使验证集不平衡，也能显著提升最差组准确率。
- 适合关注模型公平性与鲁棒性的研究者使用。
最后一层重训练（LLR）方法——在标准经验风险最小化训练后，对神经网络最后一层进行重新初始化并在保留集上重新训练——因其能有效缓解对虚假相关性的依赖并提升少数群体性能而受到关注。令人惊讶的是，即使保留集是训练集的不平衡子集，LLR仍可提升最差组准确率。我们最初假设其有效性源于保留集缓解了神经坍缩现象，从而让梯度下降的隐式偏差有利于鲁棒性。但实证研究并不支持该假设。相反，我们提供了有力证据表明，LLR的成功主要归因于保留集中更好的组别平衡。最后，我们展示近期算法CB-LLR和AFR通过隐式组平衡实现鲁棒性提升。
原文摘要 · Abstract (English)
Last-layer retraining (LLR) methods -- wherein the last layer of a neural network is reinitialized and retrained on a held-out set following ERM training -- have garnered interest as an efficient approach to rectify dependence on spurious correlations and improve performance on minority groups. Surprisingly, LLR has been found to improve worst-group accuracy even when the held-out set is an imbalanced subset of the training set. We initially hypothesize that this ``unreasonable effectiveness'' of LLR is explained by its ability to mitigate neural collapse through the held-out set, resulting in the implicit bias of gradient descent benefiting robustness. Our empirical investigation does not support this hypothesis. Instead, we present strong evidence for an alternative hypothesis: that the success of LLR is primarily due to better group balance in the held-out set. We conclude by showing how the recent algorithms CB-LLR and AFR perform implicit group-balancing to elicit a robustness improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。