改进了影响函数计算中海森矩阵逆向量积的超参数设置方法。
Revisiting inverse Hessian vector products for calculating influence functions
- 基于海森矩阵谱特性动态设定超参数，提升稳定性。
- 实验证明批量大小要求较宽松，适合大模型应用。
- 为大模型影响函数计算提供了可落地的工程方案。
影响函数是用于将模型输出归因于训练数据的常用工具。传统方法依赖于逆海森矩阵-向量积（iHVP）的计算，但经典求解器LiSSA（Agarwal et al., 2017）常因计算成本高和超参数调优复杂而被认为在大模型上不实用。本文发现，三个关键超参数——缩放因子、批量大小和迭代步数——可依据海森矩阵的谱特性（尤其是迹和最大特征值）进行设定。通过随机投影（random sketching, Swartworth and Woodruff, 2023）评估，我们发现批量大小需足够大以保证LiSSA收敛，但在所有测试模型中该要求均较温和。通过与近似布雷格曼重训练函数（PBRF, Bae et al., 2022）的对比，我们实证验证了上述结论。最后，论文讨论了逆海森矩阵在影响函数计算中的核心作用。
原文摘要 · Abstract (English)
Influence functions are a popular tool for attributing a model's output to training data. The traditional approach relies on the calculation of inverse Hessian-vector products (iHVP), but the classical solver "Linear time Stochastic Second-order Algorithm" (LiSSA, Agarwal et al. (2017)) is often deemed impractical for large models due to expensive computation and hyperparameter tuning. We show that the three hyperparameters -- the scaling factor, the batch size, and the number of steps -- can be chosen depending on the spectral properties of the Hessian, particularly its trace and largest eigenvalue. By evaluating with random sketching (Swartworth and Woodruff, 2023), we find that the batch size has to be sufficiently large for LiSSA to converge; however, for all of the models we consider, the requirement is mild. We confirm our findings empirically by comparing to Proximal Bregman Retraining Functions (PBRF, Bae et al. (2022)). Finally, we discuss what role the inverse Hessian plays in calculating the influence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。