提出随机投影保留影响函数的统一理论,指导高效计算。
A Unified Theory of Random Projection for Influence Functions
- 基于投影保持性分析,给出精确保留条件。
- 揭示正则化可降低投影门槛,由有效维度决定。
- 支持分块投影与复杂曲率结构,适合实际应用。
影响函数及相关数据归因得分形式为 $g^{ op}F^{-1}g^{ ext{′}}$,其中 $F\succeq 0$ 为曲率算子。在现代过参数化模型中,形成或求逆 $F\in\mathbb{R}^{d\times d}$ 成本过高,常通过随机投影构造小规模投影 $P \in \mathbb{R}^{m\times d}$ 实现可扩展计算。现有方法多依赖 Johnson--Lindenstrauss (JL) 引理,但该引理不涉及投影在求逆时的行为。本文发展统一理论,刻画投影在何种条件下能保证影响函数的保真性。当 $g,g^\prime\in\text{range}(F)$ 时:1)无正则化投影下,精确保留成立当且仅当 $P$ 在 $\text{range}(F)$ 上为单射,需 $m\geq \text{rank}(F)$;2)正则化投影中,岭正则化根本改变投影约束,近似保证由 $F$ 在正则化尺度下的有效维度决定;3)对 Kronecker 分解曲率 $F=A\otimes E$,即使 $P=P_A\otimes P_E$ 存在行相关性(违反 i.i.d. 假设),其保证仍成立。超出该范围时,分析测试梯度位于 $\ker(F)$ 的情形,量化泄漏项,给出一般测试点的影响查询保证。整体工作建立新理论,明确投影保真条件,并提供实践中选择投影大小的合理依据。
原文摘要 · Abstract (English)
Influence functions and related data attribution scores take the form of $g^{\top}F^{-1}g^{\prime}$, where $F\succeq 0$ is a curvature operator. In modern overparametrized models, forming or inverting $F\in\mathbb{R}^{d\times d}$ is prohibitive, motivating scalable influence computation via random projection with a sketch $P \in \mathbb{R}^{m\times d}$. This practice is commonly justified via the Johnson--Lindenstrauss (JL) lemma, which ensures approximate preservation of Euclidean geometry for a fixed dataset. However, JL does not address how sketching behaves under inversion. Furthermore, there is no existing theory that explains how sketching interacts with other widely-used techniques, such as ridge regularization and structured curvature approximations. We develop a unified theory characterizing when projection provably preserves influence functions. When $g,g^{\prime}\in\text{range}(F)$, we show that: 1) Unregularized projection: exact preservation holds iff $P$ is injective on $\text{range}(F)$, which necessitates $m\geq \text{rank}(F)$; 2) Regularized projection: ridge regularization fundamentally alters the sketching barrier, with approximation guarantees governed by the effective dimension of $F$ at the regularization scale; 3) Factorized influence: for Kronecker-factored curvatures $F=A\otimes E$, the guarantees continue to hold for decoupled sketches $P=P_A\otimes P_E$, even though such sketches exhibit row correlations that violate i.i.d. assumptions. Beyond this range-restricted setting, we analyze out-of-range test gradients and quantify a leakage term that arises when test gradients have components in $\ker(F)$. This yields guarantees for influence queries on general test points. Overall, this work develops a novel theory that characterizes when projection provably preserves influence and provides principled guidance for choosing the sketch size in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。