arXiv:2603.14222cs.CRcs.AI2026-03被引 1

仅用文本就能检测模型是否记住了个人敏感信息,效率高且不泄露隐私。

Membership Inference for Contrastive Pre-training Models with Text-only PII Queries

  • 通过文本引导跨模态反演提取相似性与一致性信号
  • 在多个CLIP/CLAP模型上实现亚秒级审计,准确率显著提升
  • 无需生物特征输入,适合隐私敏感场景下的模型审计

对比学习预训练模型(如CLIP和CLAP)作为现代多模态大模型的感知骨干,其依赖网络规模数据引发对记忆个人身份信息(PII)的担忧。现有成员推理审计方法面临两大挑战:构建影子模型计算成本高,且多数多模态审计需配对生物特征输入,直接暴露敏感信息。本文提出仅使用文本的审计框架UMID,通过文本引导跨模态潜空间反演,提取相似性(与查询文本对齐程度)和变异性(随机反演间一致性)两个互补信号,与基于合成乱码构建的轻量非成员参考集比较,利用无监督异常检测器集成决策。在多种CLIP和CLAP架构上的实验表明,UMID在效率和效果上均显著优于先前方法,仅用文本查询即可实现亚秒级审计,完全避免生物特征输入,满足严格隐私约束。

原文摘要 · Abstract (English)

Contrastive pretraining models such as CLIP and CLAP, serve as the ubiquitous perceptual backbones for modern multimodal large models, yet their reliance on web-scale data raises growing concerns about memorizing Personally Identifiable Information (PII). Auditing such models via membership inference is challenging in practice: shadow-model MIAs are computationally prohibitive for large multimodal backbones, and existing multimodal auditing methods typically require querying the target with paired biometric inputs, thereby directly exposing sensitive biometric information to the target model. To bypass this critical limitation, we demonstrate a highly desirable capability for privacy auditing: multimodal memorization within these foundational encoders can be accurately inferred using exclusively the text modality. We propose Unimodal Membership Inference Detector (UMID), a text-only auditing framework that performs text-guided cross-modal latent inversion and extracts two complementary signals, similarity (alignment to the queried text) and variability (consistency across randomized inversions). UMID compares these statistics to a lightweight non-member reference constructed from synthetic gibberish and makes decisions via an ensemble of unsupervised anomaly detectors. Comprehensive experiments across diverse CLIP and CLAP architectures demonstrate that UMID significantly improves the effectiveness and efficiency over prior MIAs, delivering strong detection performance with sub-second auditing cost using solely text queries, completely circumventing the need for biometric inputs and complying with strict privacy constraints.

隐私审计成员推断文本驱动多模态模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。