用编码器模拟解码器梯度影响，快速实现大规模数据溯源。
Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution
- 将解码器梯度影响转化为编码器可学习的特征表示
- 在百万级数据上实现毫秒级响应，效率提升10倍以上
- 适合需快速定位有毒/偏见样本的数据清洗场景
随着大语言模型能力的提升，通过筛选训练数据来构建高质量数据集的需求日益增长。数据归属（Data Attribution, DA）方法旨在评估训练集中单个样本对模型生成特定输出的预置作用。例如，可识别导致模型产生有害行为的训练样本来源。现有方法多基于影响函数范式量化这种预置效应，但其处理速度慢、存储开销大，难以在大规模数据集上实际应用。本文提出 Influcoder，一种高效且低成本的影响函数型数据归属方法，通过将解码器梯度影响排名压缩为编码器表示，实现在大规模数据上的快速推理，显著降低计算与存储成本。
原文摘要 · Abstract (English)
With the growth of LLMs' (Large Language Models) capabilities, there has been an increasing push to curate high quality datasets by filtering samples in the training data. In general, Data Attribution (DA) methods aim to estimate how individual samples in a training dataset can precondition a model to generate certain outputs. As an example, one might be interested in which samples in the data could be the source of toxic behavior after training the LLM. Many methods quantify this conditioning through the paradigm of influence functions. While methods of this family are effective in its function, they lack the necessary processing speed and storage compactness to be practically implemented on large datasets. We propose a method, Influcoder, as a quick and cost-effective approach to influence-based Data Attribution at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。