用弱监督分离提示中演示与查询的潜在变化，提升大模型少样本学习的稳定性与效率。
Disentangling Latent Shifts of In-Context Learning with Weak Supervision
- 以ICL生成伪标签，通过轻量适配器分离演示与查询的潜在影响
- 在噪声教师输出下仍优于教师，实现伪标签纠错与覆盖扩展
- 显著提升任务泛化性、稳定性和推理效率，适合新演示快速适配
上下文学习（ICL）使大语言模型通过提示中的标注示例实现少样本学习。尽管灵活，但随着示例增多，其性能易出现不稳定。为此，我们将ICL视为弱监督来源，提出一种参数高效的方法，将演示引发的潜在变化与查询本身的变化分离。基于ICL的教师模型为未标注查询生成伪标签，学生模型仅依赖查询输入预测这些标签，并通过轻量适配器更新。该方法将演示效应压缩为紧凑可复用的表示，支持高效推理并可与新演示组合使用。尽管训练数据来自噪声教师输出，学生模型常通过伪标签修正与覆盖扩展优于教师，符合弱到强的泛化规律。实验表明，该方法在域内与域外任务上均提升了泛化性、稳定性和效率，优于标准ICL及已有解耦方法。
原文摘要 · Abstract (English)
In-context learning (ICL) enables large language models to perform few-shot learning by conditioning on labeled examples in the prompt. Despite its flexibility, ICL suffers from instability -- especially as prompt length increases with more demonstrations. To address this, we treat ICL as a source of weak supervision and propose a parameter-efficient method that disentangles demonstration-induced latent shifts from those of the query. An ICL-based teacher generates pseudo-labels on unlabeled queries, while a student predicts them using only the query input, updating a lightweight adapter. This captures demonstration effects in a compact, reusable form, enabling efficient inference while remaining composable with new demonstrations. Although trained on noisy teacher outputs, the student often outperforms its teacher through pseudo-label correction and coverage expansion, consistent with the weak-to-strong generalization effect. Empirically, our method improves generalization, stability, and efficiency across both in-domain and out-of-domain tasks, surpassing standard ICL and prior disentanglement methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。