用稀疏化技术加速梯度数据归因，大幅降低大模型计算开销。
GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projection
- 通过梯度稀疏化与投影压缩，实现亚线性时空复杂度。
- 在十亿级模型上速度提升最高达165%，影响评估精度不变。
- 适合需要高效分析训练样本贡献的大模型研究者使用。
基于梯度的数据归因方法（如影响函数）对于理解单个训练样本的影响至关重要，无需重复训练模型。然而，其可扩展性常受每样本梯度计算带来的高计算与内存成本限制。本文提出GraSS，一种新型梯度压缩算法及其针对线性层的变体FactGraSS，显式利用每样本梯度的固有稀疏性，实现亚线性空间与时间复杂度。大量实验表明该方法有效，在保持数据影响保真度的同时获得显著加速。特别地，FactGraSS在十亿级模型上相比先前最先进基线速度提升高达165%。代码已公开于https://github.com/TRAIS-Lab/GraSS。
原文摘要 · Abstract (English)
Gradient-based data attribution methods, such as influence functions, are critical for understanding the impact of individual training samples without requiring repeated model retraining. However, their scalability is often limited by the high computational and memory costs associated with per-sample gradient computation. In this work, we propose GraSS, a novel gradient compression algorithm and its variants FactGraSS for linear layers specifically, that explicitly leverage the inherent sparsity of per-sample gradients to achieve sub-linear space and time complexity. Extensive experiments demonstrate the effectiveness of our approach, achieving substantial speedups while preserving data influence fidelity. In particular, FactGraSS achieves up to 165% faster throughput on billion-scale models compared to the previous state-of-the-art baselines. Our code is publicly available at https://github.com/TRAIS-Lab/GraSS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。