用专家混合机制提升大模型少样本学习效果，更省内存且抗干扰。
Mixtures of In-Context Learners
- 将演示文本分组为专家，通过学习加权融合其输出。
- 在7个数据集上5个超越基线，最多提升13%，且用更少演示更快推理。
- 对噪声、不平衡或域外数据更鲁棒，适合实际应用中的不完美数据。
上下文学习（ICL）通过提供示例让大语言模型适应任务，无需微调参数；但其无法区分示例质量，且导致Transformer模型复杂度随示例数平方增长，耗尽上下文内存。为此，我们提出混合上下文学习者（MoICL），将示例子集视为专家，学习一个加权函数以基于训练集合并它们的输出分布。实验显示，在7个分类数据集中有5个表现优于强基线（相比ICL和LENS最高提升13%）。同时，MoICL在保持性能前提下减少所需演示数量并降低推理时间，改善了ICL的帕累托前沿。此外，该方法对域外（最高+11%）、不平衡（最高+49%）或含噪声（最高+38%）示例更具鲁棒性，甚至能过滤不良示例。总体而言，MoICL是一种更高效、表达力更强的示例学习方式，不耗尽上下文窗口或内存。
原文摘要 · Abstract (English)
In-context learning (ICL) adapts LLMs by providing demonstrations without fine-tuning the model parameters; however, it does not differentiate between demonstrations and quadratically increases the complexity of Transformer LLMs, exhausting the memory. As a solution, we propose Mixtures of In-Context Learners (MoICL), a novel approach to treat subsets of demonstrations as experts and learn a weighting function to merge their output distributions based on a training set. In our experiments, we show performance improvements on 5 out of 7 classification datasets compared to a set of strong baselines (up to +13\% compared to ICL and LENS). Moreover, we enhance the Pareto frontier of ICL by reducing the inference time needed to achieve the same performance with fewer demonstrations. Finally, MoICL is more robust to out-of-domain (up to +11\%), imbalanced (up to +49\%), or noisy demonstrations (up to +38\%) or can filter these out from datasets. Overall, MoICL is a more expressive approach to learning from demonstrations without exhausting the context window or memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。