用负相关点过程构建更小的采样核心集,理论证明优于独立采样。
Small coresets via negative dependence: DPPs, linear statistics, and concentration
- 将核心集损失视为点过程的线性统计量,建立理论分析框架。
- 首次证明基于DPP的核心集规模可显著小于独立采样,且精度不降。
- 适用于向量目标函数,拓展了核心集在复杂任务中的应用范围。
确定性点过程(DPPs)具有可调节的负相关性,其采样可高效进行,是子采样任务(如小批量选择或核心集构建)的理想候选。核心集是从大规模训练集中选取的子集,使得在该子集上最小化经验损失,可作为原始全集损失最小化的可控替代。通常,这种控制表现为:核心集上的平均损失在参数空间中均匀逼近总损失。近期工作提供了使用DPP构建随机核心集的有力实证支持,并给出了一些有启发性的理论结果,但仍有关键问题未解,尤其是基于DPP的核心集基数是否本质上小于独立采样所构造的核心集。本文正面回答此问题,证明了DPP可严格优于独立采样。我们提出将核心集损失视为(随机)核心集的线性统计量,从而将核心集问题转化为对DPP线性统计量集中性的更一般问题。我们建立了有效的集中不等式,其适用范围远超现有成果,涵盖一般非投影甚至非对称核函数。此类核最近在机器学习中受到关注,但理论工具匮乏,本工作为此提供了新支持。此外,我们首次解决了向量值目标函数的核心集问题,为该领域带来新突破。
原文摘要 · Abstract (English)
Determinantal point processes (DPPs) are random configurations of points with tunable negative dependence. Because sampling is tractable, DPPs are natural candidates for subsampling tasks, such as minibatch selection or coreset construction. A \emph{coreset} is a subset of a (large) training set, such that minimizing an empirical loss averaged over the coreset is a controlled replacement for the intractable minimization of the original empirical loss. Typically, the control takes the form of a guarantee that the average loss over the coreset approximates the total loss uniformly across the parameter space. Recent work has provided significant empirical support in favor of using DPPs to build randomized coresets, coupled with interesting theoretical results that are suggestive but leave some key questions unanswered. In particular, the central question of whether the cardinality of a DPP-based coreset is fundamentally smaller than one based on independent sampling remained open. In this paper, we answer this question in the affirmative, demonstrating that \emph{DPPs can provably outperform independently drawn coresets}. In this vein, we contribute a conceptual understanding of coreset loss as a \emph{linear statistic} of the (random) coreset. We leverage this structural observation to connect the coresets problem to a more general problem of concentration phenomena for linear statistics of DPPs, wherein we obtain \emph{effective concentration inequalities that extend well-beyond the state-of-the-art}, encompassing general non-projection, even non-symmetric kernels. The latter have been recently shown to be of interest in machine learning beyond coresets, but come with a limited theoretical toolbox, to the extension of which our result contributes. Finally, we are also able to address the coresets problem for vector-valued objective functions, a novelty in the coresets literature.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。