arXiv:2603.04981cs.AI2026-03被引 2

新方法通过动态选择更代表性的样本,实现训练加速且不损失精度。

Rethinking Representativeness and Diversity in Dynamic Data Selection

  • 用高频特征覆盖度定义代表性,替代传统几何中心性。
  • 引入使用频率惩罚机制,逐步引入稀有特征以提升多样性。
  • 无需额外计算,可实现2倍以上训练加速,兼容多种模型。

动态数据选择通过持续采样数据集子集来加速训练,同时保持精度。本文重新思考样本评估中的两个核心概念:代表性与多样性。代表性不再依赖局部几何中心性,而是定义为覆盖数据集层面的常见或高频特征因子;多样性则从过程角度定义,要求选择轨迹在训练过程中逐步包含互补的稀有因子。基于此,提出一个包含三部分的动态选择框架:首先,在可插拔的特征空间中评分代表性,使用目标数据集训练的稀疏自编码器,通过稀疏单元激活总结单个样本及全集的因子统计;其次,通过稀有因子采样与使用频率惩罚结合,实现过程级多样性,可证明抑制垄断并降低梯度偏差;第三,将二维评分耦合平滑调度器,无需额外梯度、影响估计或二阶计算,实现从核心模式固化到稀有因子探索的过渡。在视觉与文本任务的五个基准上实验表明,该方法在多种模型上均实现了更优的准确率-效率权衡,相比全量数据训练,实现2倍以上加速,且精度持平或超越。代码将公开。

原文摘要 · Abstract (English)

Dynamic data selection accelerates training by sampling a changing subset of the dataset while preserving accuracy. We rethink two core notions underlying sample evaluation: representativeness and diversity. Instead of local geometric centrality, we define representativeness as coverage of dataset-level common or high-frequency feature factors. Instead of within-subset dispersion, we define diversity at the process level, requiring the selection trajectory to gradually include complementary rare factors over training. Based on this view, we propose a dynamic selection framework with three components. First, we score representativeness in a plug-in feature space to prioritize samples covering frequent factors. We instantiate this with a sparse autoencoder trained on the target dataset, using sparse unit activations to summarize both individual samples and dataset-wide factor statistics. Second, we realize process-level diversity by combining rare-factor sampling with a Usage-Frequency Penalty that promotes sample rotation, provably discourages monopoly, and reduces gradient bias. Third, we couple the two-dimensional scoring with a smooth scheduler that transitions selection from core-pattern consolidation to rare-factor exploration, without extra gradients, influence estimates, or second-order computations on the training model. Extensive experiments on five benchmarks across vision and text tasks demonstrate improved accuracy-efficiency trade-offs across models. Our method matches or exceeds full-data accuracy with over 2x training acceleration. Code will be released.

数据选择训练加速多样性稀疏编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。