arXiv:2510.11832cs.LG2025-10被引 1

提出零阶近似方法,高效估算训练数据对模型的影响。

Z0-Inf: Zeroth Order Approximation for Data Influence

  • 仅用中间检查点的损失值,无需梯度与海森矩阵计算。
  • 在大语言模型上自影响估计精度优于现有方法。
  • 适用于不可导损失函数,适合数据质量评估与调试。

理解单个训练样本对模型预测行为的影响是分析和改进现代机器学习系统的关键。估计这种影响可支持数据选择与模型调试等应用,尤其自影响(即训练点对自身的影响力)在数据质量评估与异常检测中广泛应用。然而,现有方法因准确率低或计算成本高,难以用于大规模模型:多数方法近似效果差,或依赖难以扩展的梯度与逆海森矩阵计算。本文提出一种高效的零阶近似方法,仅需少量时间和内存即可估算训练数据影响。该方法仅依赖训练与测试数据在中间检查点的损失值及检查点本身,适用于不可导损失函数。在大语言模型上,其自影响估计精度显著优于现有方法,且训练-测试影响估计精度相当或更优,实现了可扩展、实用的训练数据影响分析。

原文摘要 · Abstract (English)

A critical aspect of analyzing and improving modern machine learning systems lies in understanding how individual training examples influence a model's predictive behavior. Estimating this influence enables critical applications, including data selection and model debugging; in particular, self-influence, which quantifies the influence of a training point on itself, has found many uses in data quality assessment and outlier detection. Existing methods for measuring data influence, however, are often impractical for large models due to low accuracy or prohibitive computational costs: most approaches either provide poor approximations or rely on gradients and inverse-Hessian computations that remain challenging to scale. In this work, we introduce a highly efficient zeroth-order approximation for estimating the influence of training data that requires only a fraction of the time and memory footprint of prior methods. Notably, our method relies solely on loss values of intermediate checkpoints on the training and test data, along with the checkpoints themselves, making it broadly applicable even when the loss function of interest is non-differentiable. Beyond its computational efficiency, our approach achieves superior accuracy in estimating self-influence and comparable or improved accuracy in estimating train-test influence for fine-tuned large language models, enabling scalable and practical analysis of how training data shapes model behavior.

数据影响零阶近似大模型分析自影响

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。