arXiv:2503.00441cs.CVcs.LG2025-03CVPR被引 3

提出分片适配方法,保护数据与模型隐私的同时实现高效微调。

Split Adaptation for Pre-trained Vision Transformers

  • 将ViT拆分为前后端,仅共享低比特量化前端以保护模型
  • 客户端添加双层噪声并用增强技术抗噪,支持少样本适配
  • 兼顾数据防泄露、模型不外泄,客户端计算开销极低

视觉变换器(ViTs)在大规模数据上预训练后,已成为基础模型的重要组成部分,可在下游任务中通过极少调整获得优异性能。然而,在隐私敏感领域,客户端通常不愿共享数据,现有适配方法多需直接访问数据,难以适用。简单方案是将预训练ViT发送至客户端本地微调,但存在模型知识产权泄露风险且增加客户端计算负担。为此,本文提出一种新的分片适配(Split Adaptation, SA)方法,在保护数据与模型的前提下实现高效下游适配。SA受分片学习启发,将预训练ViT分为前端和后端,仅共享前端用于提取数据表征。与常规分片学习不同,SA采用低比特量化参数,防止模型直接暴露;同时,客户端对前端及提取表征添加双层噪声,确保数据安全。为缓解噪声带来的性能下降,SA引入数据级与模型级的分布外增强,并采用补丁检索增强策略减轻过拟合。在多个数据集上的大量实验表明,SA优于当前最优方法,在抵御高级数据重构攻击的同时,几乎不增加客户端计算成本。

原文摘要 · Abstract (English)

Vision Transformers (ViTs), extensively pre-trained on large-scale datasets, have become essential to foundation models, allowing excellent performance on diverse downstream tasks with minimal adaptation. Consequently, there is growing interest in adapting pre-trained ViTs across various fields, including privacy-sensitive domains where clients are often reluctant to share their data. Existing adaptation methods typically require direct data access, rendering them infeasible under these constraints. A straightforward solution may be sending the pre-trained ViT to clients for local adaptation, which poses issues of model intellectual property protection and incurs heavy client computation overhead. To address these issues, we propose a novel split adaptation (SA) method that enables effective downstream adaptation while protecting data and models. SA, inspired by split learning (SL), segments the pre-trained ViT into a frontend and a backend, with only the frontend shared with the client for data representation extraction. But unlike regular SL, SA replaces frontend parameters with low-bit quantized values, preventing direct exposure of the model. SA allows the client to add bi-level noise to the frontend and the extracted data representations, ensuring data protection. Accordingly, SA incorporates data-level and model-level out-of-distribution enhancements to mitigate noise injection's impact on adaptation performance. Our SA focuses on the challenging few-shot adaptation and adopts patch retrieval augmentation for overfitting alleviation. Extensive experiments on multiple datasets validate SA's superiority over state-of-the-art methods and demonstrate its defense against advanced data reconstruction attacks while preventing model leakage with minimal computation cost on the client side. The source codes can be found at https://github.com/conditionWang/Split_Adaptation.

视觉Transformer隐私保护模型压缩少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。