arXiv:2604.16197cs.LG2026-04

提出Rise方法,让大模型数据归因更高效

Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation

  • 聚焦输出层影响热点,用双通道压缩梯度信息
  • 在320亿参数模型上存储量降低112倍,支持大规模应用
  • 适合需要高效数据选优与溯源的模型开发者

数据归因与估值对理解大语言模型的数据-模型协同至关重要,但现有基于梯度的方法在大模型上面临可扩展性挑战。受人类认知启发,我们提出RISE(Readout Influence Sketching Estimator),不计算全模型梯度,而是聚焦输出层的影响热点,利用梯度的可分解外积形式,构建词汇残差通道(RH)与语义投影误差通道(GH)。通过应用CountSketch投影对两通道进行压缩,在保持归因精度的同时大幅减少存储。在OLMo(1B–32B)和Pythia(14M–6.9B)系列模型上,相比RapidIn,RISE存储压缩达112倍,并首次实现320亿参数模型上的可操作性,而梯度基方法如RapidIn与ZO-Inf已不可行。我们在两种范式下验证:(1)回溯归因——检索特定预测的影响训练样本;(2)前瞻估值——零样本评估候选数据价值。应用于三类任务:Howdy后门检测、金融-医疗领域分离、Brain Rot高质量数据筛选。在闭环Brain Rot研究中,使用RISE选出的数据持续预训练,带来稳定下游性能提升。整体而言,RISE为现代大模型提供了实用且可扩展的影响分析与训练数据选择基础工具。

原文摘要 · Abstract (English)

Data attribution and valuation are critical for understanding data-model synergy for Large Language Models (LLMs), yet existing gradient-based methods suffer from scalability challenges on LLMs. Inspired by human cognition, where decision making relies on a focused readout of relevant memories rather than replaying all pathways, we introduce RISE (Readout Influence Sketching Estimator). Instead of computing and indexing gradients across the entire LLM, RISE focuses on influence hotspots at the output layer, where influence signals concentrate, and the gradient admits a decomposed outer-product form. This enables a dual-channel representation combining a lexical residual channel (RH) and a semantic projected-error channel (GH). Applying CountSketch projections to these channels achieves strong compression while maintaining accurate attribution. Across the OLMo (1B-32B) and Pythia (14M-6.9B) families, RISE reduces index storage by up to 112$\times$ compared to RapidIn and scales to 32B parameters LLM, where gradient-based baselines such as RapidIn and ZO-Inf become memory-infeasible. We evaluate RISE on two paradigms: (1) retrospective attribution, retrieving influential training examples for specific predictions, and (2) prospective valuation, scoring candidate data utility zero-shot. We validate RISE on three tasks: Howdy backdoor data detection, Finance-Medical domain separation, and Brain Rot high-quality data selection. In a closed-loop Brain Rot study, continued pretraining on RISE-selected data yields consistent downstream improvements. Overall, RISE provides a practical and scalable primitive for influence analysis and training-data selection in modern large language models.

数据归因大模型高效算法选数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。