用信息论方法追踪模型预测的关键训练数据，提升可解释性。
A Bayesian Information-Theoretic Approach to Data Attribution
- 基于信息熵变化评估样本重要性，避免受标签噪声干扰。
- 通过高斯过程近似实现大模型下的高效计算，支持子集检索。
- 适用于大规模数据溯源，适合安全与可信AI研究者使用。
训练数据溯源（TDA）旨在将模型预测追溯到关键训练样本，以增强可解释性与安全性。本文将其建模为贝叶斯信息论问题：通过移除样本后预测熵的增加量来评分，该准则强调样本对降低预测不确定性的贡献而非标签噪声。为适配现代神经网络，我们利用切线特征构建高斯过程代理模型近似信息损失。该方法在单样本溯源中与经典影响分数一致，并在子集层面促进多样性。针对更大规模检索，进一步采用信息增益目标并引入方差修正，实现在向量数据库中的可扩展溯源。实验表明，在反事实敏感性、真实数据召回和核心集选择任务上表现优异，证明该方法可扩展至现代架构，有效连接理论与实践。
原文摘要 · Abstract (English)
Training Data Attribution (TDA) seeks to trace model predictions back to influential training examples, enhancing interpretability and safety. We formulate TDA as a Bayesian information-theoretic problem: subsets are scored by the information loss they induce - the entropy increase at a query when removed. This criterion credits examples for resolving predictive uncertainty rather than label noise. To scale to modern networks, we approximate information loss using a Gaussian Process surrogate built from tangent features. We show this aligns with classical influence scores for single-example attribution while promoting diversity for subsets. For even larger-scale retrieval, we relax to an information-gain objective and add a variance correction for scalable attribution in vector databases. Experiments show competitive performance on counterfactual sensitivity, ground-truth retrieval and coreset selection, showing that our method scales to modern architectures while bridging principled measures with practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。