用随机丢弃压缩梯度，加速大模型影响函数计算
Toward Efficient Influence Function: Dropout as a Compression Tool
- 用dropout作为梯度压缩工具，降低计算开销
- 在保持数据影响关键特征的前提下，大幅减少内存与时间消耗
- 适合需要高效分析训练数据影响的大模型研究者
评估训练数据对机器学习模型的影响对于理解模型行为、提升透明度和优化训练数据选择至关重要。影响函数为量化单个训练数据点对特定测试样本性能的影响提供了理论框架。然而，即使采用近似方法，影响函数的计算仍面临显著的计算与内存开销，因为涉及的梯度大小与模型本身相当。本文提出一种新方法，利用随机丢弃（dropout）作为梯度压缩机制，更高效地计算影响函数。该方法不仅在影响函数计算阶段显著降低资源消耗，同时在梯度压缩过程中也实现高效处理。通过理论分析与实证验证，我们证明该方法能保留数据影响的关键成分，使现代大规模模型中的影响函数应用成为可能。
原文摘要 · Abstract (English)
Assessing the impact the training data on machine learning models is crucial for understanding the behavior of the model, enhancing the transparency, and selecting training data. Influence function provides a theoretical framework for quantifying the effect of training data points on model's performance given a specific test data. However, the computational and memory costs of influence function presents significant challenges, especially for large-scale models, even when using approximation methods, since the gradients involved in computation are as large as the model itself. In this work, we introduce a novel approach that leverages dropout as a gradient compression mechanism to compute the influence function more efficiently. Our method significantly reduces computational and memory overhead, not only during the influence function computation but also in gradient compression process. Through theoretical analysis and empirical validation, we demonstrate that our method could preserves critical components of the data influence and enables its application to modern large-scale models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。