用多样性采样降低分布对齐的噪声,提升模型泛化能力。
Improving Distribution Alignment with Diversity-based Sampling
- 通过k-DPP或k-means++实现多样化小批量采样
- 减少分布距离估计误差,提升域外准确率
- 适用于生物声学事件检测等真实场景
领域偏移在机器学习中普遍存在,部署到真实数据时会显著降低模型性能。分布对齐方法旨在通过最小化不同领域间分布差异来学习不变特征表示,但基于随机梯度下降(SGD)训练时,差异估计值可能极为嘈杂,且子群体比例变化会导致领域错配,阻碍方法收益。本文提出通过在每个小批量中引入多样性来改进这些估计,同时平衡数据并降低梯度方差,从而增强模型泛化能力。文中设计了两种基于k-确定性点过程(k-DPP)和k-means++算法的多样性采样器,可作为标准随机采样的即插即用替代方案。在真实世界领域偏移任务——生物声学事件检测中,实验表明:1)所生成的小批量更代表全数据集;2)在给定样本量下,降低分布间距离估计误差;3)提升两种分布对齐算法及标准ERM的域外准确率。
原文摘要 · Abstract (English)
Domain shifts are ubiquitous in machine learning, and can substantially degrade a model's performance when deployed to real-world data. To address this, distribution alignment methods aim to learn feature representations which are invariant across domains, by minimising the discrepancy between the distributions. However, the discrepancy estimates can be extremely noisy when training via stochastic gradient descent (SGD), and shifts in the relative proportions of different subgroups can lead to domain misalignments; these can both stifle the benefits of the method. This paper proposes to improve these estimates by inducing diversity in each sampled minibatch. This simultaneously balances the data and reduces the variance of the gradients, thereby enhancing the model's generalisation ability. We describe two options for diversity-based data samplers, based on the k-determinantal point process (k-DPP) and the k-means++ algorithm, which can function as drop-in replacements for a standard random sampler. On a real-world domain shift task of bioacoustic event detection, we show that both options 1) yield minibatches which are more representative of the full dataset; 2) reduce the distance estimation error between distributions, for a given sample size; and 3) improve out-of-distribution accuracy for two distribution alignment algorithms, as well as standard ERM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。