arXiv:2410.02268cs.LGcs.AI2024-10ICLR被引 6

用结构熵选样本,兼顾全局和局部信息,提升学习效率与效果。

Structural-Entropy-Based Sample Selection for Efficient and Effective Learning

  • 基于样本相似性构建kNN图,用结构熵量化全局信息。
  • 结合结构熵与训练难度,评估样本重要性并采样。
  • 适用于监督、主动及持续学习,提升代表性与多样性。

样本选择通过提供有信息量且具代表性的样本,提升机器学习模型的效率与效果。通常可将样本建模为样本图,节点为样本,边表示其相似性。现有方法多依赖局部信息(如样本训练难度),忽视了全局信息(如连通模式),导致选择不优,因全局信息对保持图的结构特性至关重要。为此,本文采用结构熵量化全局信息,并利用Shapley值将其无损分解至各节点。基于此,提出结构熵-样本选择(SES)方法,融合全局与局部信息以选取有信息量且具代表性的样本。SES首先基于样本相似性构建kNN图,再结合结构熵(全局指标)与训练难度(局部指标)衡量样本重要性,最后采用重要性加权的蓝噪声采样策略,选出多样且具代表性的样本集。在监督学习、主动学习与持续学习三种场景的全面实验,均验证了该方法的有效性。

原文摘要 · Abstract (English)

Sample selection improves the efficiency and effectiveness of machine learning models by providing informative and representative samples. Typically, samples can be modeled as a sample graph, where nodes are samples and edges represent their similarities. Most existing methods are based on local information, such as the training difficulty of samples, thereby overlooking global information, such as connectivity patterns. This oversight can result in suboptimal selection because global information is crucial for ensuring that the selected samples well represent the structural properties of the graph. To address this issue, we employ structural entropy to quantify global information and losslessly decompose it from the whole graph to individual nodes using the Shapley value. Based on the decomposition, we present $\textbf{S}$tructural-$\textbf{E}$ntropy-based sample $\textbf{S}$election ($\textbf{SES}$), a method that integrates both global and local information to select informative and representative samples. SES begins by constructing a $k$NN-graph among samples based on their similarities. It then measures sample importance by combining structural entropy (global metric) with training difficulty (local metric). Finally, SES applies importance-biased blue noise sampling to select a set of diverse and representative samples. Comprehensive experiments on three learning scenarios -- supervised learning, active learning, and continual learning -- clearly demonstrate the effectiveness of our method.

样本选择结构熵kNN图高效学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。