arXiv:2606.05165cs.LGcs.CL2026-06

通过激活空间稀疏恢复,快速定位大模型训练数据的贡献度。

STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations

论文配图:STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
图 1 · 摘自论文原文
  • 在激活空间建模数据影响,用轻量级控制算子模拟数据增删效果。
  • 实现比现有方法快13倍的训练数据溯源,准确率达当前最优。
  • 适用于数据筛选、污染检测等实际场景,适合关注模型可解释性的研究者。

训练数据溯源(TDA)旨在将模型预测回溯至其训练数据。传统金标准依赖因果干预,通过增删数据观察模型变化,但对大语言模型重复训练成本过高。现有方法多在参数空间近似梯度效应,不仅计算开销巨大,且依赖局部近似。本文提出新思路:不追踪参数变化,而是建模训练数据在激活空间的功能影响。提出STRIDE(基于引导的训练数据影响分解)框架,将TDA建模为压缩感知风格的稀疏恢复问题。该框架学习轻量级“引导算子”,模拟数据子集带来的行为改变。通过测量这些算子对测试预测的扰动,利用稀疏线性分解恢复单个训练样本的影响。STRIDE在大模型预训练溯源任务中达到当前最优性能,速度比之前方法快13倍。进一步通过数据选择、数据污染检测和定性分析验证其实际价值。

原文摘要 · Abstract (English)

Training Data Attribution (TDA) seeks to trace a model's predictions back to its training data. The gold standard for TDA relies on causal interventions, observing how a model changes when data is added or removed, but repeated retraining is computationally challenging for Large Language Models (LLMs). Consequently, most approaches approximate this effect in the parameter space using gradients. However, tracking gradients across billions of parameters is not only prohibitively expensive but relies on local approximations. In this work, we propose a shift: rather than estimating parameter changes, we model the functional effect of training data in the activation space. We introduce STRIDE (Steering-based Training Data Influence Decomposition), a framework that formulates TDA as a sparse recovery problem in the spirit of compressive sensing. STRIDE learns lightweight "steering operators" that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, we recover individual training example influences via sparse linear decomposition. STRIDE achieves state-of-the-art for LLM pre-training attribution while being an order of magnitude ($13\times$) faster than previous art. We further validate its practical utility through downstream applications including data selection, data contamination, and qualitative analysis.

模型可解释性数据溯源稀疏恢复大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。