用中间特征直接恢复语义信息,揭示视觉语言模型隐私漏洞
CapRecover: A Cross-Modality Feature Inversion Attack Framework on Vision Language Models
- 跨模态反演框架,跳过图像重建直接提取标签和描述
- 在CIFAR-10上标签准确率达92.71%,COCO2017生成句式流畅度达ROUGE-L 0.52
- 发现深层卷积层语义信息更强,提出加噪-去噪防护方案
随着视觉语言模型(VLMs)在分部式DNN架构中广泛应用——视觉编码器(如ResNet、ViT)运行于用户设备,将中间特征发送至云端——从这些中间特征泄露语义信息的隐私风险日益加剧。现有图像重建方法常产生模糊、语义不清的结果。为直接应对语义泄露问题,本文提出CapRecover,一种跨模态反演框架,可直接从中间特征恢复高层语义内容(如标签或标题),无需重建图像。我们在多个数据集和目标模型上评估该框架,结果表明其在语义恢复方面表现优异:在CIFAR-10上达到最高92.71%的Top-1标签准确率;在COCO2017上,基于ResNet50特征生成的标题具有高达0.52的ROUGE-L得分。分析进一步显示,深层卷积层比浅层编码了显著更多的语义信息。为缓解语义泄露,我们提出一种简单有效的防护策略:在每层中间特征添加随机噪声,并在下一层去除该噪声。实验表明,该方法可在无需额外训练成本的前提下有效防止语义泄露。
原文摘要 · Abstract (English)
As Vision-Language Models (VLMs) are increasingly deployed in split-DNN configurations--with visual encoders (e.g., ResNet, ViT) operating on user devices and sending intermediate features to the cloud--there is a growing privacy risk from semantic information leakage. Existing approaches to reconstructing images from these intermediate features often result in blurry, semantically ambiguous images. To directly address semantic leakage, we propose CapRecover, a cross-modality inversion framework that recovers high-level semantic content, such as labels or captions, directly from intermediate features without image reconstruction. We evaluate CapRecover on multiple datasets and victim models, demonstrating strong performance in semantic recovery. Specifically, CapRecover achieves up to 92.71% Top-1 label accuracy on CIFAR-10 and generates fluent captions from ResNet50 features on COCO2017 with ROUGE-L scores up to 0.52. Our analysis further reveals that deeper convolutional layers encode significantly more semantic information compared to shallow layers. To mitigate semantic leakage, we introduce a simple yet effective protection method: adding random noise to intermediate features at each layer and removing the noise in the next layer. Experimental results show that this approach prevents semantic leakage without additional training costs. Our code is available at https://jus1mple.github.io/Image2CaptionAttack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。