动态筛选数据，训练提速近45%且性能不降。
Data-Efficient Training by Evolved Sampling
- 根据损失变化动态选样，支持灵活频率调节。
- 在多个任务上节省近45%训练时间，性能不变。
- 可插拔式设计，适合大规模模型加速。
数据选择旨在加速学习并保持性能。核心思路是识别对训练有显著贡献的样本。本文提出一种简单有效的动态采样框架——进化采样（Evolved Sampling, ES），基于批次级损失动态和增强的损失差值进行数据选择，支持灵活频率调节，显著减少反向传播时间，同时保持模型性能。由于结构简洁，该方法可轻松扩展为结合集级采样（形成ESWP）以进一步加速。作为即插即用框架，ES(WP)在多种预训练与微调任务中均实现无损加速，最高节省近45%实际运行时间。结果推动了对现代大规模机器学习数据效率的深入研究。
原文摘要 · Abstract (English)
Data selection is designed to accelerate learning with preserved performance. To achieve this, a fundamental thought is to identify informative data samples with significant contributions to the training. In this work, we propose \textbf{Evolved Sampling} (\textbf{ES}), a simple yet effective framework for \emph{dynamic} sampling along the training process. This method conducts \em batch \em level data selection based on the dynamics of losses and augmented \emph{loss differences}, which enables flexible \emph{frequency tuning}, and hence significantly reduces the back propagation time with maintained model performance. Due to its conciseness, ES is also readily extensible to incorporate \em set \em level data selection (to form ES with pruning, \textbf{ESWP}) for further accelerations. As a plug-and-play framework, ES(WP) consistently achieves lossless training accelerations across various pre-training and post-training tasks, saving up to nearly 45\% wall-clock time. Our results motivate further investigations on the data efficiency aspect of modern large-scale machine learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。