首个实现多模态上下文学习的差分隐私框架，支持大量样本且保证严格隐私。
Differentially Private Multimodal In-Context Learning
- 将多模态演示压缩为激活空间中的紧凑任务向量，降低隐私成本。
- ε=1.0时在VizWiz上达50%准确率，远超零样本（35%）和非私有方法（55%）。
- 适用于医疗影像等敏感场景，适合需保护数据隐私的研究者使用。
视觉语言模型正被应用于医疗影像、个人照片等敏感领域，但现有差分隐私的上下文学习方法仅限于少样本、纯文本场景，因隐私代价随处理令牌数增长而受限。本文提出首个支持多模态、多样本上下文学习的差分隐私框架——差分隐私多模态任务向量（DP-MTV）。该方法将私有数据划分为不重叠块，对每层进行裁剪以控制敏感度，并在聚合时添加校准噪声，仅需一次噪声注入即可支持无限次推理查询。在三种VLM架构的八个基准上评估，支持有或无辅助数据部署。在ε=1.0下，于VizWiz任务中取得50%准确率，优于零样本（35%）与非私有基线（55%），在强隐私约束下仍保留大部分上下文学习收益。
原文摘要 · Abstract (English)
Vision-language models are increasingly applied to sensitive domains such as medical imaging and personal photographs, yet existing differentially private methods for in-context learning are limited to few-shot, text-only settings because privacy cost scales with the number of tokens processed. We present Differentially Private Multimodal Task Vectors (DP-MTV), the first framework enabling many-shot multimodal in-context learning with formal $(\varepsilon, δ)$-differential privacy by aggregating hundreds of demonstrations into compact task vectors in activation space. DP-MTV partitions private data into disjoint chunks, applies per-layer clipping to bound sensitivity, and adds calibrated noise to the aggregate, requiring only a single noise addition that enables unlimited inference queries. We evaluate on eight benchmarks across three VLM architectures, supporting deployment with or without auxiliary data. At $\varepsilon=1.0$, DP-MTV achieves 50% on VizWiz compared to 55% non-private and 35% zero-shot, preserving most of the gain from in-context learning under meaningful privacy constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。