通过束搜索优化上下文示例顺序，提升大模型少样本学习效果
Learning to Search Effective Example Sequences for In-Context Learning
- 联合考虑示例序列长度、组成和排列，用束搜索逐步构建最优序列
- 在多个数据集上显著提升大模型少样本学习性能，最高提升12.3%
- 适合需要高效利用少量标注数据的场景，如低资源语言任务
大语言模型具备出色的少样本学习能力，但其性能受上下文示例序列的影响较大。影响因素包括序列长度、组成与排列方式，以及与具体查询的相关性。现有方法通常孤立处理这些因素，忽视了它们之间的相互依赖关系。此外，示例序列的搜索空间庞大，阻碍了整体优化方法的发展。本文提出基于束搜索的示例序列构造器（BESC），在推理过程中联合考虑所有关键因素，逐步构建最优序列。该设计使束搜索能有效压缩搜索空间。在多个数据集和语言模型上的实验表明，性能有显著提升。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate impressive few-shot learning capabilities, but their performance varies widely based on the sequence of in-context examples. Key factors influencing this include the sequence's length, composition, and arrangement, as well as its relation to the specific query. Existing methods often tackle these factors in isolation, overlooking their interdependencies. Moreover, the extensive search space for selecting optimal sequences complicates the development of a holistic approach. In this work, we introduce Beam Search-based Example Sequence Constructor (BESC), a novel method for learning to construct optimal example sequences. BESC addresses all key factors involved in sequence selection by considering them jointly during inference, while incrementally building the sequence. This design enables the use of beam search to significantly reduce the complexity of the search space. Experiments across various datasets and language models show notable improvements in performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。