arXiv:2608.02690cs.LG2026-08

通过轨迹对齐与结构化稀疏优化,高效选出能保持训练性能的小型核心数据集。

GLOBE: Trajectory-Aligned Gradient Matching with Structured SparseOptimization for Coreset Selection

论文配图:GLOBE: Trajectory-Aligned Gradient Matching with Structured SparseOptimization for Coreset Selection
图 1 · 摘自论文原文
  • 用多阶段梯度轨迹表征样本,捕捉其在训练中的动态影响。
  • 在低保留率下仍显著提升测试准确率,6个基准上均优于现有方法。
  • 适合资源受限场景下的高效模型训练,尤其适用于小样本学习。

设备端深度神经网络训练受大规模数据集计算与内存开销的制约。核心数据集选择通过保留少量真实训练样本提供实用解决方案。然而,现有基于梯度的方法通常依赖单一模型快照的梯度,并采用贪心或追捕式选择过程,难以捕捉演变的优化动态并处理强相关样本。我们提出 GLOBE(梯度局部平衡提取),一种轨迹对齐的核心数据集选择框架,将样本选择建模为全局优化的稀疏加权问题。GLOBE 通过多个训练检查点构建的梯度轨迹表示每个样本,从而捕捉其在整个优化过程中的影响。为保留全数据集的训练行为,我们引入多阶匹配目标,联合对齐梯度轨迹的一阶均值和投影无中心二阶矩。GLOBE 还结合组 LASSO、弹性网络正则化及非负预算约束,诱导组级与样本级稀疏性,同时稳定相关轨迹的权重。最后,类别平衡的 Top-K 选择在有限采样预算下保证类别覆盖。在六个基准和五种评估架构上的实验表明,GLOBE 在下游测试准确率上持续优于现有核心数据集选择方法,尤其在低保留率时表现突出。结果凸显了动态梯度信息、多阶分布匹配与结构化稀疏性结合在数据高效学习中的有效性。

原文摘要 · Abstract (English)

On-device training of deep neural networks is fundamentally constrained by the computational and memory costs of large-scale datasets. Coreset selection offers a practical solution by retaining only a compact subset of real training samples. However, existing gradient-based methods commonly rely on gradients computed at a single model snapshot and employ greedy or pursuit-based selection procedures, limiting their ability to capture evolving optimization dynamics and handle strongly correlated samples. We propose GLOBE (Gradient Local-Balanced Extraction), a trajectory-aligned coreset selection framework that formulates sample selection as a globally optimized sparse weighting problem. GLOBE represents each sample by a gradient trajectory constructed across multiple training checkpoints, thereby capturing its influence throughout different stages of optimization. To preserve the training behavior of the full dataset, we introduce a multi-order matching objective that jointly aligns the first-order mean and projected uncentered second-order moments of gradient trajectories. GLOBE further combines Group LASSO, Elastic Net regularization, and nonnegative budget constraints to induce group- and sample-level sparsity while stabilizing the weights of correlated trajectories. Finally, class-balanced Top-K selection maintains adequate category coverage under limited sampling budgets. Experiments across six benchmarks and five evaluation architectures demonstrate that GLOBE consistently outperforms existing coreset selection methods in downstream test accuracy, particularly at low retention ratios. These results highlight the effectiveness of combining dynamic gradient information, multi-order distribution matching, and structured sparsity for data-efficient learning.

核心数据集稀疏优化梯度轨迹小样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。