提出分布式数据归因，揭示影响函数的深层机制
Distributional Training Data Attribution: What do Influence Functions Sample?
- 构建分布式数据归因框架，分析训练多样性对模型输出的影响
- 发现影响函数本质是无展开微分的极限形式，无需凸性假设
- 适用于视觉变换器和扩散模型的数据重要性分析
深度学习训练固有的随机性（如初始化和批量采样）会导致相同数据集训练出不同模型，但传统数据归因方法未充分考虑此问题。本文提出分布式训练数据归因（d-TDA），旨在预测模型输出在多次训练中的分布如何依赖于数据集。研究发现，流行的影响力函数（IFs）实际上是‘隐含分布式’的：它们可被推导为无展开微分的极限形式，且无需严格的凸性假设。这一发现为影响力函数在深度学习中的有效性提供了新视角。实验表明，d-TDA在视觉变换器的数据剪枝和扩散模型中识别关键样本方面具有实际应用价值。
原文摘要 · Abstract (English)
Randomness is an unavoidable part of training deep learning models, yet something that traditional training data attribution algorithms fail to rigorously account for. They ignore the fact that, due to stochasticity in the initialisation and batching, training on the same dataset can yield different models. In this paper, we address this shortcoming through introducing distributional training data attribution (d-TDA), the goal of which is to predict how the distribution of model outputs (over training runs) depends upon the dataset. Intriguingly, we find that influence functions (IFs), a popular data attribution tool, are 'secretly distributional': they emerge from our framework as the limit to unrolled differentiation, without requiring restrictive convexity assumptions. This provides a new perspective on the effectiveness of IFs in deep learning. We demonstrate the practical utility of d-TDA in experiments, including improving data pruning for vision transformers and identifying influential examples with diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。