通过双向梯度优化，定位大模型输出对应的训练数据来源。
Data Attribution in Large Language Models via Bidirectional Gradient Optimization
- 用反向梯度优化模拟输出对训练数据的影响
- 在多个预训练模型上优于现有影响度量方法
- 适合需要可解释性的AI治理与数据溯源场景
大型语言模型（LLM）日益应用于各类场景，引发治理、问责与数据溯源的关键问题。理解哪些训练数据最影响模型输出仍是一个基础性难题。本文针对自回归LLM提出训练数据溯源（TDA），基于逆向建模：若模型在训练时见过生成的输出，训练数据会如何变化？通过在生成文本样本上使用双向梯度优化（梯度上升与下降），测量各训练样本损失的变化，实现任意粒度的数据溯源，支持事实与风格双重溯源。我们在已知数据集的预训练模型上评估该方法，结果表明其在影响度量方面超越现有基线，显著提升模型可解释性，满足可问责AI系统的核心需求。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed across diverse applications, raising critical questions for governance, accountability, and data provenance. Understanding which training data most influenced a model's output remains a fundamental open problem. We address this challenge through training data attribution (TDA) for auto-regressive LLMs by expanding upon the inverse formulation: How would training data be affected if the model had seen the generated output during training? Our method perturbs the base model using bidirectional gradient optimization (gradient ascent and descent) on a generated text sample and measures the resulting change in loss across training samples. Our framework supports attribution at arbitrary data granularity, enabling both factual and stylistic attribution. We evaluate our method against baselines on pretrained models with known datasets, and show that it outperforms previous work on influence metrics, thereby enhancing model interpretability, an essential requirement for accountable AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。