提升训练数据归因精度，关键在更准计算逆海森向量积。
Better Training Data Attribution via Better Inverse Hessian-Vector Products
- 用EKFAC预条件+尼曼级数迭代，高效逼近逆海森向量积。
- 相比传统方法减少迭代次数，归因准确率显著提升。
- 适合需要精准数据溯源的模型可解释性研究者使用。
训练数据归因（TDA）可揭示哪些训练数据导致了模型行为。基于梯度的TDA方法如影响函数和展开微分均涉及类似逆海森向量积（iHVP）的计算，该计算难以高效近似。本文提出ASTRA算法，利用EKFAC预条件加速尼曼级数迭代，实现更精确的iHVP近似。ASTRA易于调参，迭代次数更少，且精度优于基于EKFAC的现有方法。实验表明，提升iHVP近似精度能显著改善TDA性能。
原文摘要 · Abstract (English)
Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is difficult to approximate efficiently. We introduce an algorithm (ASTRA) which uses the EKFAC-preconditioner on Neumann series iterations to arrive at an accurate iHVP approximation for TDA. ASTRA is easy to tune, requires fewer iterations than Neumann series iterations, and is more accurate than EKFAC-based approximations. Using ASTRA, we show that improving the accuracy of the iHVP approximation can significantly improve TDA performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。