提出首个验证掩码模型是否训练过特定数据集的方法。
Dataset Ownership Verification for Pre-trained Masked Models
- 基于掩码重建难度差异设计验证机制
- 在ImageNet-1K和WikiText-103上显著优于现有方法
- 适合数据集版权保护与模型溯源场景
高质量开源数据集是推动深度学习快速发展的关键,但也面临被滥用的风险。数据集所有权验证成为保护权益的重要手段,但现有方法主要针对监督模型和对比预训练模型,难以适用于日益流行的掩码模型。本文提出首个针对掩码建模的数据集所有权验证方法DOV4MM,旨在判断可疑黑盒模型是否在特定无标签数据集上进行过预训练。该方法基于实证观察:若模型在目标数据集上预训练,其嵌入空间中掩码信息的重建难度会显著不同于未预训练的模型。我们在10个掩码图像模型(ImageNet-1K)和4个掩码语言模型(WikiText-103)上验证了DOV4MM的有效性,结果表明其能显著拒绝原假设(p值远低于0.05),性能全面超越已有方法。代码已公开于https://github.com/xieyc99/DOV4MM。
原文摘要 · Abstract (English)
High-quality open-source datasets have emerged as a pivotal catalyst driving the swift advancement of deep learning, while facing the looming threat of potential exploitation. Protecting these datasets is of paramount importance for the interests of their owners. The verification of dataset ownership has evolved into a crucial approach in this domain; however, existing verification techniques are predominantly tailored to supervised models and contrastive pre-trained models, rendering them ill-suited for direct application to the increasingly prevalent masked models. In this work, we introduce the inaugural methodology addressing this critical, yet unresolved challenge, termed Dataset Ownership Verification for Masked Modeling (DOV4MM). The central objective is to ascertain whether a suspicious black-box model has been pre-trained on a particular unlabeled dataset, thereby assisting dataset owners in safeguarding their rights. DOV4MM is grounded in our empirical observation that when a model is pre-trained on the target dataset, the difficulty of reconstructing masked information within the embedding space exhibits a marked contrast to models not pre-trained on that dataset. We validated the efficacy of DOV4MM through ten masked image models on ImageNet-1K and four masked language models on WikiText-103. The results demonstrate that DOV4MM rejects the null hypothesis, with a $p$-value considerably below 0.05, surpassing all prior approaches. Code is available at https://github.com/xieyc99/DOV4MM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。