让模型更精准识别训练数据影响，效率比传统方法高近百倍。
Enhancing Training Data Attribution with Representational Optimization
- 用可训练编码器和注意力池化学习专用于数据溯源的表征。
- 在指令微调大模型上性能接近梯度法，推理速度提升近100倍。
- 适合需要高效、可靠数据溯源的AI系统研发与审计场景。
训练数据溯源（TDA）旨在衡量训练数据对模型预测的影响。基于梯度的方法（如影响函数）虽有理论基础，但计算成本高，难以大规模应用；而基于表示的方法虽可扩展，却通常依赖未针对溯源优化的启发式嵌入，限制了准确性。为此，我们提出AirRep，一种可扩展的表示型方法，通过学习任务特定且与模型对齐的表征，显式优化以提升TDA精度。AirRep引入两项关键创新：一个针对溯源质量可训练的编码器，以及基于注意力的池化机制，实现群体影响的精准估计。通过在自动构建的训练子集上使用排名目标进行训练，这些子集由其对目标预测的实际影响标注。在指令微调的大语言模型上实验表明，AirRep性能媲美最先进的梯度方法，同时推理效率提高近两个数量级。进一步分析显示其在不同任务和模型间具备鲁棒性与泛化能力。代码已开源：https://github.com/sunnweiwei/AirRep。
原文摘要 · Abstract (English)
Training data attribution (TDA) methods aim to measure how training data impacts a model's predictions. While gradient-based attribution methods, such as influence functions, offer theoretical grounding, their computational costs make them impractical for large-scale applications. Representation-based approaches are far more scalable, but typically rely on heuristic embeddings that are not optimized for attribution, limiting their fidelity. To address these challenges, we propose AirRep, a scalable, representation-based approach that closes this gap by learning task-specific and model-aligned representations optimized explicitly for TDA. AirRep introduces two key innovations: a trainable encoder tuned for attribution quality, and an attention-based pooling mechanism that enables accurate estimation of group-wise influence. We train AirRep using a ranking objective over automatically constructed training subsets labeled by their empirical effect on target predictions. Experiments on instruction-tuned LLMs demonstrate that AirRep achieves performance on par with state-of-the-art gradient-based approaches while being nearly two orders of magnitude more efficient at inference time. Further analysis highlights its robustness and generalization across tasks and models. Our code is available at https://github.com/sunnweiwei/AirRep
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。