用提示学习保护视觉语言模型的知识产权,防止未经授权的数据域迁移。
Vision-Language Model IP Protection via Prompt-based Learning
- 基于CLIP框架,通过提示学习融合图像风格与内容信息构建版权屏障。
- 在授权与非授权数据域间实现特征阻断,性能下降可控,准确率损失低于5%。
- 适合需要模型部署权限控制的AI企业或研究机构使用。
视觉语言模型(如CLIP)在视觉识别中表现卓越,推动了对其知识产权(IP)保护的需求。现有方法多依赖视觉主干网络,语义表达不足。为此,本文提出IP-CLIP,一种轻量级的版权保护策略,利用冻结的CLIP视觉主干提取图像风格与内容信息,并融入提示学习中,形成有效屏障,阻止授权域特征向未授权域转移。此外,设计风格增强分支,构建授权与非授权域的特征库,融合自增强与跨域特征,进一步强化阻断能力。最后,提出三项新指标,平衡授权与非授权域的性能损失。大量实验表明,该方法在多种场景下均具应用潜力。
原文摘要 · Abstract (English)
Vision-language models (VLMs) like CLIP (Contrastive Language-Image Pre-Training) have seen remarkable success in visual recognition, highlighting the increasing need to safeguard the intellectual property (IP) of well-trained models. Effective IP protection extends beyond ensuring authorized usage; it also necessitates restricting model deployment to authorized data domains, particularly when the model is fine-tuned for specific target domains. However, current IP protection methods often rely solely on the visual backbone, which may lack sufficient semantic richness. To bridge this gap, we introduce IP-CLIP, a lightweight IP protection strategy tailored to CLIP, employing a prompt-based learning approach. By leveraging the frozen visual backbone of CLIP, we extract both image style and content information, incorporating them into the learning of IP prompt. This strategy acts as a robust barrier, effectively preventing the unauthorized transfer of features from authorized domains to unauthorized ones. Additionally, we propose a style-enhancement branch that constructs feature banks for both authorized and unauthorized domains. This branch integrates self-enhanced and cross-domain features, further strengthening IP-CLIP's capability to block features from unauthorized domains. Finally, we present new three metrics designed to better balance the performance degradation of authorized and unauthorized domains. Comprehensive experiments in various scenarios demonstrate its promising potential for application in IP protection tasks for VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。