用专家模型乘积框架实现高效私有上下文学习，准确率提升超30%。
Private PoEtry: Private In-Context Learning via Product of Experts
- 基于专家模型乘积重构私有上下文学习，理论基础扎实。
- 在五类任务上平均准确率提升超30个百分点，优于现有方法。
- 适合需保护隐私又想快速适应新任务的研究者使用。
上下文学习（ICL）使大语言模型在推理时仅通过少量示例即可适应新任务，避免任务特定微调。然而，上下文示例可能包含敏感信息，需防止模型输出泄露。现有差分隐私（DP）方法或计算开销大，或依赖效果有限的启发式手段，如上下文过采样、合成数据生成或不必要的阈值处理。本文从专家模型乘积视角重新构建私有ICL，提供理论基础，并可轻松并行化。我们在文本分类、数学和视觉-语言五个数据集上评估该方法，结果表明，相比先前的DP-ICL方法，本方法平均准确率提升超过30个百分点，同时保持强隐私保障。
原文摘要 · Abstract (English)
In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks with only a small set of examples at inference time, thereby avoiding task-specific fine-tuning. However, in-context examples may contain privacy-sensitive information that should not be revealed through model outputs. Existing differential privacy (DP) approaches to ICL are either computationally expensive or rely on heuristics with limited effectiveness, including context oversampling, synthetic data generation, or unnecessary thresholding. We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized. We evaluate our method across five datasets in text classification, math, and vision-language. We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。