arXiv:2606.24774cs.CV2026-06

通过分析模型参数梯度,精准识别视觉语言大模型是否泄露训练数据。

Revealing Training Data Exposure in Vision Language Large Models via Parameter Gradients

论文配图:Revealing Training Data Exposure in Vision Language Large Models via Parameter Gradients
图 1 · 摘自论文原文
  • 基于梯度稳定性检测训练数据,不依赖输出结果
  • 在医疗和通用数据集上均显著优于现有方法
  • 适合关注模型版权与数据隐私的研究者

视觉-语言大模型(VLLMs)在大规模爬取语料上训练,引发严峻的版权与数据溯源问题,尤其在医疗领域,患者医学图像与临床报告需严格隐私保护。现有训练数据检测方法在跨模态场景中表现不佳,或依赖表面输出信号,判别力不足。我们提出GradAudit,一种基于梯度的审计框架,考察模型内部优化动态而非将其视为黑箱。核心观察为:模型参数收敛至训练样本梯度稳定且对齐的区域,而非训练样本则梯度嘈杂不一致。通过分析这些梯度特征,GradAudit能有效区分真实图像-文本关联,而非单一模态归属。实验证明,在医疗与通用数据集上,该方法在预训练与微调阶段均显著优于当前最优基线。案例研究显示,现有方法不仅低估了未经授权数据的使用程度,且随着模型越新越先进,低估现象越严重。

原文摘要 · Abstract (English)

Vision-Language Large Models (VLLMs) trained on massive crawled corpora raise pressing copyright and data-provenance concerns. These concerns are particularly acute in healthcare, where patient medical images paired with clinical reports demand rigorous privacy safeguards. However, existing training data detection methods either fail in cross-modal scenarios or rely on superficial output signals with insufficient discriminative power. We introduce GradAudit, a gradient-based auditing framework that examines internal optimization dynamics rather than treating VLLMs as black boxes. Our approach builds on a key observation: model parameters converge to regions where gradients on training samples become stable and well-aligned, whereas gradients on non-training samples remain noisy and inconsistent. By analyzing these gradient signatures, GradAudit achieves strong separability and detects genuine image-text associations learned during training, not merely individual modality membership. Empirically, across both medical and general-domain datasets, GradAudit substantially outperforms state-of-the-art baselines in both pretraining and fine-tuning VLLMs. In a case study employing copyrighted content, we show that existing training data detection methods not only underestimate the extent of unauthorized data usage, but that this underestimation becomes more pronounced as models become more recent and more advanced.

模型审计数据隐私视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。