优化重要性权重以应对子群体分布偏移，提升模型泛化能力。
Optimizing importance weighting in the presence of sub-population shifts
- 基于偏差-方差权衡，提出双层优化框架同时调整权重与模型参数。
- 在有限训练样本下，优化权重使测试误差显著降低，提升泛化性能。
- 适用于存在子群体分布偏移的深度神经网络微调场景。
训练数据与测试数据之间的分布偏移会严重损害机器学习模型的性能。重要性加权通过在训练中为不同数据点分配不同权重来缓解此问题。我们指出，现有启发式权重确定方法次优，因其忽略了训练样本有限时估计模型方差的增加。本文从偏差-方差权衡角度解释最优权重，并提出一种双层优化过程，同时优化权重与模型参数。我们将该优化应用于现有重要性加权技术，在子群体分布偏移下对深层神经网络进行最后一层重训练，实验表明优化权重能显著提升泛化性能。
原文摘要 · Abstract (English)
A distribution shift between the training and test data can severely harm performance of machine learning models. Importance weighting addresses this issue by assigning different weights to data points during training. We argue that existing heuristics for determining the weights are suboptimal, as they neglect the increase of the variance of the estimated model due to the finite sample size of the training data. We interpret the optimal weights in terms of a bias-variance trade-off, and propose a bi-level optimization procedure in which the weights and model parameters are optimized simultaneously. We apply this optimization to existing importance weighting techniques for last-layer retraining of deep neural networks in the presence of sub-population shifts and show empirically that optimizing weights significantly improves generalization performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。