arXiv:2508.20230cs.LG2025-08被引 4

用损失轨迹相关性选关键数据,高效且稳定。

Coresets from Trajectories: Selecting Data via Correlation of Loss Differences

  • 基于验证集损失轨迹的差异相关性筛选核心数据集。
  • 在CIFAR-100和ImageNet上,小样本下性能接近顶尖方法,误差<1%。
  • 跨模型、早期快照可用,无需分层采样,适合资源受限场景。

深度学习模型虽在各领域表现优异,但在实时或资源受限场景面临可扩展性挑战。为此,我们提出损失差异相关性(CLD),一种简单高效的核子集选择度量,通过衡量训练样本与保留验证集损失轨迹的一致性,识别最具影响力的训练样本。CLD仅需在训练检查点计算样本级损失值,避免了高成本的梯度与曲率计算。我们构建了一个通用理论框架,证明了基于CLD的核心集收敛误差被样本选择与验证集代表性之间的对齐程度所上界控制。在CIFAR-100和ImageNet-1k上,CLD核心集在不同子集大小下通常优于或接近现有最优方法,即使未领先也保持在更昂贵基线的1%以内。该方法在不同架构(ResNet、VGG、DenseNet)间迁移效果良好,代理到目标选择时退化低于1%。此外,仅使用早期检查点即可保持稳定性,精度损失可忽略。最后,通过类内验证对齐,CLD具备内在偏差抑制能力,无需额外分层采样。这些特性使CLD成为可扩展数据优化中原理清晰、高效、稳定且可迁移的工具。

原文摘要 · Abstract (English)

Deep learning models achieve state-of-the-art performance across domains but face scalability challenges in real-time or resource-constrained scenarios. To address this, we propose Correlation of Loss Differences (CLD), a simple and scalable metric for coreset selection that identifies the most impactful training samples by measuring their alignment with the loss trajectories of a held-out validation set. CLD is highly efficient, requiring only per-sample loss values computed at training checkpoints, and avoiding the costly gradient and curvature computations used in many existing subset selection methods. We develop a general theoretical framework that establishes convergence guarantees for CLD-based coresets, demonstrating that the convergence error is upper-bounded by the alignment of the selected samples and the representativeness of the validation set. On CIFAR-100 and ImageNet-1k, CLD-based coresets typically outperform or closely match state-of-the-art methods across subset sizes, and remain within 1% of more computationally expensive baselines even when not leading. CLD transfers effectively across architectures (ResNet, VGG, DenseNet), enabling proxy-to-target selection with <1% degradation. Moreover, CLD is stable when using only early checkpoints, incurring negligible accuracy loss. Finally, CLD exhibits inherent bias reduction via per-class validation alignment, obviating the need for additional stratified sampling. Together, these properties make CLD a principled, efficient, stable, and transferable tool for scalable dataset optimization.

核心集高效训练数据选择可扩展性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。