用批量损失均值推算样本重要性,轻松实现动态数据剪枝。
Batch Loss Score for Dynamic Data Pruning
- 用指数移动平均的批量损失作为样本得分,无需计算每样本损失。
- 在14个数据集、18个模型上无损剪枝20%-50%样本。
- 代码仅需三行注入,适合复杂模型中难以获取单样本损失的场景。
动态数据剪枝通过训练时选择性跳过信息量少的样本加速深度学习。尽管单样本损失是常见的重要度指标,但在复杂模型或损失函数下获取困难,常需大量实现工作。本文提出批量损失评分(BLS),利用易于获取的批量损失的指数移动平均(EMA)为每个样本赋分。从单一样本视角看,批量损失可视为其缩放后个体损失的带噪测量,噪声源于批次的随机组成。理论证明EMA相当于一阶低通滤波器,有效抑制批次组成带来的高频噪声。由此得到的分数近似反映样本对损失的平滑持续贡献,为BLS作为样本重要性代理提供理论支持。BLS具有极简代码集成(仅需三行注入),并可一键替换现有基于单样本损失的方法。实验表明,其在14个数据集、11项任务、18种模型上实现无损剪枝20%-50%,展现出广泛适用性,尤其适用于难以访问单样本损失的复杂场景。代码已公开于https://github.com/mrazhou/BLS。
原文摘要 · Abstract (English)
Dynamic data pruning accelerates deep learning by selectively omitting less informative samples during training. While per-sample loss is a common importance metric, obtaining it can be challenging or infeasible for complex models or loss functions, often requiring significant implementation effort. This work proposes the Batch Loss Score (BLS), a computationally efficient alternative using an Exponential Moving Average (EMA) of readily available batch losses to assign scores to individual samples. We frame the batch loss, from the perspective of a single sample, as a noisy measurement of its scaled individual loss, with noise originating from stochastic batch composition. It is formally shown that the EMA mechanism functions as a first-order low-pass filter, attenuating high-frequency batch composition noise. This yields a score approximating the smoothed and persistent contribution of the individual sample to the loss, providing a theoretical grounding for BLS as a proxy for sample importance. BLS demonstrates remarkable code integration simplicity (\textbf{three-line injection}) and readily adapts existing per-sample loss-based methods (\textbf{one-line proxy}). Its effectiveness is demonstrated by enhancing two such methods to losslessly prune \textbf{20\%-50\%} of samples across \textit{14 datasets}, \textit{11 tasks} and \textit{18 models}, highlighting its utility and broad applicability, especially for complex scenarios where per-sample loss is difficult to access. Code is available at https://github.com/mrazhou/BLS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。