无需反向传播即可实现高效数据溯源,适合大规模模型实时审计。
Scalable Data Attribution via Forward-Only Test-Time Inference
- 训练时用短程梯度传播模拟数据影响,推理时仅需前向计算
- 在MLP基准上优于或媲美TRAK方法,推理成本降低数量级
- 适合大模型部署场景,如语言模型对特定数据源的溯源
数据溯源旨在将模型行为回溯至塑造它的训练样本,支持调试、审计和数据估值。经典影响函数方法虽有理论基础,但因推理时需昂贵的反向传播或海森矩阵求逆,难以应用于现代网络。本文提出一种新方法,在保持一阶反事实目标的同时,消除了每个查询的反向传播需求。该方法在训练阶段通过短时梯度传播模拟每个训练样本的参数影响,并在后续通过仅前向评估读取任意查询的溯源结果。此设计将计算从推理转移至模拟,契合实际部署场景:模型可能服务数十亿用户查询,却源自有限的数据源(如大型语言模型基于多样化语料训练,但需追踪特定出版商如纽约时报的贡献)。实验表明,在标准MLP基准上,该估计器在LOO和LDS等指标上达到或超越当前最优基线TRAK,同时推理开销降低数量级。该方法结合影响函数的准确性与一阶可扩展性,为大规模预训练模型的实时数据溯源提供了理论框架。
原文摘要 · Abstract (English)
Data attribution seeks to trace model behavior back to the training examples that shaped it, enabling debugging, auditing, and data valuation at scale. Classical influence-function methods offer a principled foundation but remain impractical for modern networks because they require expensive backpropagation or Hessian inversion at inference. We propose a data attribution method that preserves the same first-order counterfactual target while eliminating per-query backward passes. Our approach simulates each training example's parameter influence through short-horizon gradient propagation during training and later reads out attributions for any query using only forward evaluations. This design shifts computation from inference to simulation, reflecting real deployment regimes where a model may serve billions of user queries but originate from a fixed, finite set of data sources (for example, a large language model trained on diverse corpora while compensating a specific publisher such as the New York Times). Empirically, on standard MLP benchmarks, our estimator matches or surpasses state-of-the-art baselines such as TRAK on standard attribution metrics (LOO and LDS) while offering orders-of-magnitude lower inference cost. By combining influence-function fidelity with first-order scalability, our method provides a theoretical framework for practical, real-time data attribution in large pretrained models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。