提出通用数据影响函数,让复杂模型也能快速分析数据重要性。
A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
- 基于统计学通用公式,推导出适用于非可分解损失的新方法。
- 在三种任务中计算速度比重训快千倍,结果接近精确值。
- 适合处理对比学习、排序等复杂任务,无需手动推导损失导数。
影响函数源自稳健统计学,近年被用于机器学习中的数据归属分析——量化单个训练样本对模型预测的影响。然而,现有方法仅适用于可分解为单样本损失之和的损失函数(如M-估计器),限制了其在对比学习、排序损失等非可分解损失场景下的应用。本文重新审视稳健统计中的通用影响函数公式,提出一种新型方法:通用影响函数(VIF),可直接应用于任意非可分解损失的模型训练。与传统统计方法相比,VIF充分利用自动微分,避免了针对每种损失函数的特殊推导。我们在三个案例中验证:生存分析的Cox回归、网络节点嵌入、列表式学习排序。结果显示,VIF估算的影响值与暴力留一法重训练结果高度一致,但计算速度提升高达10^3倍。VIF为多种机器学习范式提供了高效的数据归属分析工具,具有广泛实际应用潜力。
原文摘要 · Abstract (English)
Influence function, a technique rooted in robust statistics, has been adapted in modern machine learning for a novel application: data attribution -- quantifying how individual training data points affect a model's predictions. However, the common derivation of influence functions in the data attribution literature is limited to loss functions that can be decomposed into a sum of individual data point losses, with the most prominent examples known as M-estimators. This restricts the application of influence functions to more complex learning objectives, which we refer to as non-decomposable losses, such as contrastive or ranking losses, where a unit loss term depends on multiple data points and cannot be decomposed further. In this work, we bridge this gap by revisiting the general formulation of influence function from robust statistics, which extends beyond M-estimators. Based on this formulation, we propose a novel method, the Versatile Influence Function (VIF), that can be straightforwardly applied to machine learning models trained with any non-decomposable loss. In comparison to the classical approach in statistics, the proposed VIF is designed to fully leverage the power of auto-differentiation, hereby eliminating the need for case-specific derivations of each loss function. We demonstrate the effectiveness of VIF across three examples: Cox regression for survival analysis, node embedding for network analysis, and listwise learning-to-rank for information retrieval. In all cases, the influence estimated by VIF closely resembles the results obtained by brute-force leave-one-out retraining, while being up to $10^3$ times faster to compute. We believe VIF represents a significant advancement in data attribution, enabling efficient influence-function-based attribution across a wide range of machine learning paradigms, with broad potential for practical use cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。