用高效采样策略选最优示例，大幅减少大模型调用次数。
Sample Efficient Demonstration Selection for In-Context Learning
- 将示例选择建模为带探索的优化问题，只关注候选集合。
- 相比现有方法，减少87%调用次数，提速7倍，性能不降。
- 适合需要快速构建提示且资源受限的场景。
基于大语言模型的上下文学习在自然语言处理中广泛应用。在有限上下文长度下，挑选少数高质量示例对构建有效提示至关重要。本文将示例选择问题建模为寻找前m个最优臂的挑战性任务，面临需评估指数级数量臂的难题。提出CASE（Challenger Arm Sampling for Exemplar selection）策略，维护一个“挑战者”臂的短名单，在每轮仅选取一个臂进行评估，显著降低样本复杂度和大模型调用次数。同时引入参数化线性评分函数，构建随机线性多臂老虎机框架。实验表明，CASE在不牺牲性能的前提下，实现最高7倍的运行效率提升，大模型调用次数减少87%。代码与数据已开源。
原文摘要 · Abstract (English)
The in-context learning paradigm with LLMs has been instrumental in advancing a wide range of natural language processing tasks. The selection of few-shot examples (exemplars / demonstration samples) is essential for constructing effective prompts under context-length budget constraints. In this paper, we formulate the exemplar selection task as a top-m best arms identification problem. A key challenge in this setup is the exponentially large number of arms that need to be evaluated to identify the m-best arms. We propose CASE (Challenger Arm Sampling for Exemplar selection), a novel sample-efficient selective exploration strategy that maintains a shortlist of "challenger" arms, which are current candidates for the top-m arms. In each iteration, only one of the arms from this shortlist or the current topm set is pulled, thereby reducing sample complexity and, consequently, the number of LLM evaluations. Furthermore, we model the scores of exemplar subsets (arms) using a parameterized linear scoring function, leading to stochastic linear bandits setting. CASE achieves remarkable efficiency gains of up to 7x speedup in runtime while requiring 7x fewer LLM calls (87% reduction) without sacrificing performance compared to state-of-the-art exemplar selection methods. We release our code and data at https://github.com/kiranpurohit/CASE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。