提出新方法，让大模型影响函数计算更快更省资源。
Extending Kernel Trick to Influence Functions
- 用双重表示法降低影响函数计算复杂度，不随模型大小增长
- 在模型远大于数据集时，计算效率提升显著
- 适合大模型分析，但仅限可线性化模型
本文提出影响函数的双重表示，其计算复杂度随数据集规模增长，而非模型规模。理论与实验均表明，当模型规模远大于数据集，或在参数空间中计算原版影响函数不可行时，该方法可高效估算数据点移除对参数、模型输出和损失的影响。然而，该方法仅适用于行为可被线性近似的模型（即线性可化模型），且需存储一个大小与模型输出维度和数据集规模乘积成正比的矩阵。
原文摘要 · Abstract (English)
In this paper, we present a dual representation of the influence functions, whose computational complexity scales with dataset size rather than model size. Both analytically and experimentally, we show that this representation can be an efficient alternative to the original influence functions for estimating changes in parameters, model outputs and loss due to data point removal, when model size is large relative to dataset size, or when evaluating the original influence functions in parameter space is infeasible. The dual representation, however, is limited to linearizable models, which are models whose behavior can be approximated by their linearizations throughout training, and requires materializing a matrix, whose size grows with the product of model output dimension and dataset size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。