拆解数据剪枝中表示与选择的作用，发现表示更重要。
Disentangling the Roles of Representation and Selection in Data Pruning
- 将数据剪枝分解为表示和选择算法两部分进行系统研究
- 训练梯度等更好表示能显著提升剪枝效果，与选择算法无关
- 不同剪枝算法表现差异大，且目标不符时可能选错样本
数据剪枝通过选取小而关键的数据子集，为高效扩展NLP模型训练提供了可行路径。然而，现有方法包含多种设计选择，尚未被系统研究，限制了后续发展。本文将数据剪枝分解为数据表示和选择算法两个核心组件，并系统分析其对实例选择的影响。理论与实证结果表明：表示质量起决定性作用——如使用训练梯度作为表示，可显著提升剪枝效果，且不受选择算法影响。此外，不同选择算法在不同场景下表现各异，无一始终占优；更值得注意的是，同一目标下的算法可能选出截然不同的实例，凸显评估需谨慎。
原文摘要 · Abstract (English)
Data pruning, selecting small but impactful subsets, offers a promising way to efficiently scale NLP model training. However, existing methods often involve many different design choices, which have not been systematically studied. This limits future developments. In this work, we decompose data pruning into two key components: the data representation and the selection algorithm, and we systematically analyze their influence on the selection of instances. Our theoretical and empirical results highlight the crucial role of representations: better representations, e.g., training gradients, generally lead to a better selection of instances, regardless of the chosen selection algorithm. Furthermore, different selection algorithms excel in different settings, and none consistently outperforms the others. Moreover, the selection algorithms do not always align with their intended objectives: for example, algorithms designed for the same objective can select drastically different instances, highlighting the need for careful evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。