用多个微调的小模型提升上下文学习效果，省钱又高效。
Improving In-Context Learning with Small Language Model Ensembles
- 用多个微调小模型组成集成，增强上下文学习能力。
- 在多个自然语言理解任务上达到当前最优表现。
- 适合资源有限却需领域适配的实践者使用。
大语言模型在多种任务中表现出色，但在特定领域任务上的性能仍受限。尽管检索增强生成和微调等方法有效，但需要大量资源。上下文学习（ICL）虽成本低、效率高，但准确率难以超越先进方法。本文提出 Ensemble SuperICL，通过利用多个微调的小语言模型（SLMs）的专长来提升 ICL 性能。该方法在多个自然语言理解基准上取得当前最优结果。此外，在医疗领域标注任务中，仅使用通用语料微调的现成 SLMs 就实现了优于所有基线的大规模数据标注准确率。最后，我们进行了消融实验与敏感性分析，揭示了 Ensemble SuperICL 的作用机制。本研究响应了大模型高效领域适配的需求，为从业者提供了一种低成本、高效的解决方案。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources. In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods. We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs). Ensemble SuperICL achieves state of the art (SoTA) results on several natural language understanding benchmarks. Additionally, we test it on a medical-domain labelling task and showcase its practicality by using off-the-shelf SLMs fine-tuned on a general language task, achieving superior accuracy in large-scale data labelling compared to all baselines. Finally, we conduct an ablation study and sensitivity analyses to elucidate the underlying mechanism of Ensemble SuperICL. Our research contributes to the growing demand for efficient domain specialisation methods in LLMs, offering a cheap and effective method for practitioners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。