轻量适配器联邦大模型隐藏隐私,却可能被新攻击反推训练数据。
Reconstructing Training Data from Adapter-based Federated Large Language Models
- 基于注意力模式和梯度子空间,设计新攻击方法逆推文本。
- 在多个模型上实现99%以上召回率,大批次下仍有效。
- 揭示参数高效与隐私保护的根本矛盾,适合安全研究者参考。
基于适配器的联邦大语言模型(FedLLMs)通过冻结主干网络、仅训练低秩适配器,降低计算与通信开销,同时被认为能抑制梯度泄露并抵御现有梯度逆向攻击(GIA)。然而,我们发现低秩适配器反而引入新的可被利用的泄露通道。为此,提出面向适配器结构的无序词袋文本重建攻击(UTR),解决低维梯度、冻结主干和组合爆炸等挑战:(i) 从冻结层注意力模式推断词存在性;(ii) 在适配器梯度的低秩子空间中进行句级逆向;(iii) 利用语言先验引导约束贪婪解码以保证语义连贯。在GPT2-Large、BERT、Qwen2.5-7B及CoLA、SST-2、Rotten Tomatoes等多组实验中,UTR达到近完美重构准确率(ROUGE-1/2 > 99),即使在先前攻击失效的大批量设置下依然有效。结果揭示了参数效率与隐私保护之间的根本矛盾,挑战了‘轻量适配即更安全’的普遍认知。代码与数据见https://github.com/shwksnshwowk-wq/GIA。
原文摘要 · Abstract (English)
Adapter-based Federated Large Language Models (FedLLMs) are widely adopted to reduce the computational, storage, and communication overhead of full-parameter fine-tuning for web-scale applications while preserving user privacy. By freezing the backbone and training only compact low-rank adapters, these methods appear to limit gradient leakage and thwart existing Gradient Inversion Attacks (GIAs). Contrary to this assumption, we show that low-rank adapters create new, exploitable leakage channels. We propose the Unordered-word-bag-based Text Reconstruction (UTR) attack, a novel GIA tailored to the unique structure of adapter-based FedLLMs. UTR overcomes three core challenges: low-dimensional gradients, frozen backbones, and combinatorially large reconstruction spaces by: (i) inferring token presence from attention patterns in frozen layers, (ii) performing sentence-level inversion within the low-rank subspace of adapter gradients, and (iii) enforcing semantic coherence through constrained greedy decoding guided by language priors. Extensive experiments across diverse models (GPT2-Large, BERT, Qwen2.5-7B) and datasets (CoLA, SST-2, Rotten Tomatoes) demonstrate that UTR achieves near-perfect reconstruction accuracy (ROUGE-1/2 > 99), even with large batch size settings where prior GIAs fail completely. Our results reveal a fundamental tension between parameter efficiency and privacy in FedLLMs, challenging the prevailing belief that lightweight adaptation inherently enhances security. Our code and data are available at https://github.com/shwksnshwowk-wq/GIA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。