在数据分布变化时,选出能稳定表现的训练子集。
Distributionally Robust Coreset Selection under Covariate Shift
- 基于最坏情况误差上界,动态筛选鲁棒子集。
- 在多种分布偏移下,测试误差显著低于基线方法。
- 适合对模型泛化性要求高的实际部署场景。
核心集选择通过从原始训练数据中选取小规模子集来降低训练开销,但在实际应用中,开发阶段与部署阶段的数据分布常存在差异,且部署分布未知。为应对这一挑战,本文提出分布鲁棒核心集选择(DRCS),理论上推导了未来协变量分布偏离训练分布时的最坏情况测试误差上界。通过选择能够抑制该上界估计值的样本,实现对分布偏移的鲁棒性。该方法主要适用于凸优化训练任务,但通过适当近似也可用于深度学习。本文聚焦协变量偏移,通过实验验证了其有效性。
原文摘要 · Abstract (English)
Coreset selection, which involves selecting a small subset from an existing training dataset, is an approach to reducing training data, and various approaches have been proposed for this method. In practical situations where these methods are employed, it is often the case that the data distributions differ between the development phase and the deployment phase, with the latter being unknown. Thus, it is challenging to select an effective subset of training data that performs well across all deployment scenarios. We therefore propose Distributionally Robust Coreset Selection (DRCS). DRCS theoretically derives an estimate of the upper bound for the worst-case test error, assuming that the future covariate distribution may deviate within a defined range from the training distribution. Furthermore, by selecting instances in a way that suppresses the estimate of the upper bound for the worst-case test error, DRCS achieves distributionally robust training instance selection. This study is primarily applicable to convex training computation, but we demonstrate that it can also be applied to deep learning under appropriate approximations. In this paper, we focus on covariate shift, a type of data distribution shift, and demonstrate the effectiveness of DRCS through experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。