通过选择关键组件，比投影更高效地解释大模型训练数据影响。
Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanations
- 用贪心法选出关键模型组件构建低维表示
- 该方法在检索任务中表现优于完整梯度和随机投影
- 计算效率更高，适合实际大模型解释需求
基于梯度的大型语言模型实例解释受模型梯度高维性限制。实践中，影响估计通常仅针对参数子集以保证计算可行性，但该子集常凭直觉选取且缺乏系统评估。本文探讨：是通过选择少量架构相关组件构造低维表示，还是将完整梯度投影到低维空间更优？基于新构建的基准测试，结果表明，贪心选取的组件子集在支持训练数据检索任务方面，比完整梯度或随机投影更有效。此外，该方法在计算上比随机投影更高效，证明有目标地选择组件是实现大模型实例解释的可行策略。
原文摘要 · Abstract (English)
Gradient-based methods for instance-based explanation for large language models (LLMs) are hindered by the immense dimensionality of model gradients. In practice, influence estimation is restricted to a subset of model parameters to make computation tractable, but this subset is often chosen ad hoc and rarely justified by systematic evaluation. This paper investigates if it is better to create low-dimensional representations by selecting a small, architecturally informed subset of model components or by projecting the full gradients into a lower-dimensional space. Using a novel benchmark, we show that a greedily selected subset of components captures the information about training data influence needed for a retrieval task more effectively than either the full gradient or random projection. We further find that this approach is more computationally efficient than random projection, demonstrating that targeted component selection is a practical strategy for making instance-based explanations of large models more computationally feasible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。