arXiv:2606.16801cs.CL2026-06

用混洗技术保护大模型分治训练中的隐私,兼顾效果与安全

The Art of Mixology: Mixup-based Obfuscation for Privacy-Preserving Split Learning in Large Language Models

论文配图:The Art of Mixology: Mixup-based Obfuscation for Privacy-Preserving Split Learning in Large Language Models
图 1 · 摘自论文原文
  • 通过令牌级混淆、表征混淆和自适应梯度扰动三重机制提升隐私性
  • 在4类分类与4类生成任务中保持接近非分治训练的模型性能
  • 对前沿数据重构攻击有更强防御力,适合资源受限的私密训练场景

分治学习为资源受限用户提供了训练大语言模型的可行方案,即把计算密集层卸载到服务器,同时本地保留原始数据。然而现有隐私保护方法在性能、隐私、效率和稳定性之间难以平衡:常导致显著性能下降,仍易受先进数据重建攻击,计算通信开销过大,或在不同任务上表现不稳定。本文提出MIXGUARD,一种基于混洗的大语言模型分治学习隐私保护框架。该框架引入令牌级混淆、表征级混淆与自适应梯度扰动机制,协同作用以保留有效学习信号并防止服务器端隐私泄露。技术上,先在公开数据集上构建轻量校准模型以优化近似目标表征,再用于私有数据上的隐私保护微调。我们在四个分类任务和四个文本生成任务上,针对多种大模型家族、规模、架构和微调策略进行了广泛实验。结果表明,MIXGUARD在模型性能上接近非分治训练基线,在对抗最先进数据重建攻击方面持续优于现有分治学习防御方法,并在自适应攻击设置下保持鲁棒性。

原文摘要 · Abstract (English)

Split learning provides a practical paradigm for resource-constrained users to train Large Language Models (LLMs) by offloading computation-intensive layers to a server while keeping raw data local. However, existing privacy-preserving split learning methods still face a difficult trade-off among utility, privacy, efficiency, and stability. Specifically, these methods often suffer from substantial utility degradation, remain vulnerable to advanced data reconstruction attacks, incur prohibitive computational and communication overhead, or exhibit unstable performance across different tasks. In this paper, we propose MIXGUARD, a novel mixup-based privacy-preserving split learning framework for LLMs. MIXGUARD introduces token-level obfuscation, representation-level obfuscation, and adaptive gradient perturbation mechanisms, which operate jointly to preserve useful learning signals while preventing privacy leakage to the server. Technically, MIXGUARD first constructs a lightweight calibration model on a public dataset to refine the approximated target representation, and then applies this model during privacy-preserving fine-tuning on private data. We conduct extensive experiments on four classification tasks and four text generation tasks across multiple LLM families, model sizes, architectures, and fine-tuning strategies. The results show that MIXGUARD preserves model utility comparable to non-split training baselines, consistently achieves stronger privacy protection than existing split learning defense methods against state-of-the-art data reconstruction attacks, and remains robust under adaptive attack settings.

隐私保护分治学习大模型混洗

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。