arXiv:2603.16761cs.LGcs.CL2026-03

提出SOMP方法,高效还原大模型训练梯度中的隐私文本。

SOMP: Scalable Gradient Inversion for Large Language Models via Subspace-Guided Orthogonal Matching Pursuit

  • 将梯度反演建模为稀疏信号恢复问题,利用头级几何结构与样本稀疏性
  • 在批大小16时重建精度显著优于基线,批大小达128仍可恢复有效文本
  • 适合关注大模型训练隐私泄露风险的研究者和安全评估人员

梯度反演攻击表明,从共享梯度中可重构出私有训练文本,对大语言模型(LLM)构成隐私威胁。现有方法在小批量设置下表现良好,但面对更大批量和更长序列时因信号严重混叠、计算成本高及保真度下降而难以扩展。本文提出SOMP(子空间引导正交匹配追踪),一种可扩展的梯度反演框架,将聚合梯度中的文本恢复问题转化为稀疏信号恢复。核心洞察是:聚合的Transformer梯度保留了可利用的头级几何结构与样本级稀疏性。SOMP通过逐步缩小搜索空间并解耦混合信号,避免穷举搜索。在多个LLM家族、模型规模及五种语言上的实验表明,SOMP在聚合梯度场景下始终优于现有方法。对于批大小B=16的长序列,其重建保真度显著高于强基线,同时计算开销可控;即使在极端聚合(最高至B=128)下,仍能恢复有意义的文本,说明隐私泄露可能存在于此前攻击失效的场景中。

原文摘要 · Abstract (English)

Gradient inversion attacks reveal that private training text can be reconstructed from shared gradients, posing a privacy risk to large language models (LLMs). While prior methods perform well in small-batch settings, scaling to larger batch sizes and longer sequences remains challenging due to severe signal mixing, high computational cost, and degraded fidelity. We present SOMP (Subspace-Guided Orthogonal Matching Pursuit), a scalable gradient inversion framework that casts text recovery from aggregated gradients as a sparse signal recovery problem. Our key insight is that aggregated transformer gradients retain exploitable head-wise geometric structure together with sample-level sparsity. SOMP leverages these properties to progressively narrow the search space and disentangle mixed signals without exhaustive search. Experiments across multiple LLM families, model scales, and five languages show that SOMP consistently outperforms prior methods in the aggregated-gradient regime.For long sequences at batch size B=16, SOMP achieves substantially higher reconstruction fidelity than strong baselines, while remaining computationally competitive. Even under extreme aggregation (up to B=128), SOMP still recovers meaningful text, suggesting that privacy leakage can persist in regimes where prior attacks become much less effective.

梯度反演隐私安全大模型稀疏恢复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。