arXiv:2512.08809cs.CRcs.AI2025-12中稿 · IEEE INFOCOM 2026被引 2

PrivTune通过设备云协同实现高效私密微调,有效防御数据泄露。

PrivTune: Efficient and Privacy-Preserving Fine-Tuning of Large Language Models via Device-Cloud Collaboration

  • 在分割学习框架中注入优化噪声,使令牌模仿邻近节点特征
  • 对抗攻击成功率降至10%,性能下降仅3.33%
  • 适合需保护隐私的个性化大模型微调场景

随着大语言模型的发展,服务提供商提供模型即服务,用户可通过上传私有数据集进行定制化微调。然而,这引发了敏感数据泄露的担忧。现有方法依赖差分隐私,在设备-云协同框架中难以平衡隐私与实用性,易受推断攻击或降低微调性能。为此,本文提出PrivTune,一种基于分割学习(SL)的高效且隐私保护的微调框架。其核心思想是在SL底模的令牌表示中注入设计噪声,使每个令牌呈现$ n $-跳间接邻居特征。该过程被建模为优化问题,以计算最优噪声向量,兼顾防御与实用性目标。在此基础上,调整$d_χ$-隐私噪声分布的均值参数以匹配优化方向,并按令牌重要性缩放噪声以最小化失真。在五个数据集(涵盖分类与生成任务)上,针对三种嵌入反演攻击和三种属性推断攻击的实验表明,使用RoBERTa在Stanford Sentiment Treebank数据集上,PrivTune将攻击成功率降至10%,仅导致3.33%的性能下降,优于当前最先进基线。

原文摘要 · Abstract (English)

With the rise of large language models, service providers offer language models as a service, enabling users to fine-tune customized models via uploaded private datasets. However, this raises concerns about sensitive data leakage. Prior methods, relying on differential privacy within device-cloud collaboration frameworks, struggle to balance privacy and utility, exposing users to inference attacks or degrading fine-tuning performance. To address this, we propose PrivTune, an efficient and privacy-preserving fine-tuning framework via Split Learning (SL). The key idea of PrivTune is to inject crafted noise into token representations from the SL bottom model, making each token resemble the $n$-hop indirect neighbors. PrivTune formulates this as an optimization problem to compute the optimal noise vector, aligning with defense-utility goals. On this basis, it then adjusts the parameters (i.e., mean) of the $d_χ$-Privacy noise distribution to align with the optimization direction and scales the noise according to token importance to minimize distortion. Experiments on five datasets (covering both classification and generation tasks) against three embedding inversion and three attribute inference attacks show that, using RoBERTa on the Stanford Sentiment Treebank dataset, PrivTune reduces the attack success rate to 10% with only a 3.33% drop in utility performance, outperforming state-of-the-art baselines.

大模型微调隐私保护分割学习差分隐私

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。