用对数概率重加权提升大模型少样本学习效果，兼顾精度与效率。
Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learning
- 将长示例拆成短片段并行处理，降低内存占用
- 通过非梯度优化重加权各组输出，提升整体准确率
- 适合需要高效少样本推理的场景，如资源受限部署
在上下文学习（ICL）中，大型语言模型可通过不更新参数的方式，利用任务相关示例完成新任务适应。然而，随着示例数量增加，性能下降且计算成本呈二次增长。本文提出一种基于对数概率重加权的新框架LARA，通过将长输入示例分组为可并行的短输入，显著降低内存需求，并采用无梯度优化方法对每组输出的对数概率进行重加权融合。进一步提出二值化变体B-LARA，将权重限制为二值以简化搜索空间并过滤低信息组。在BBH和MMLU数据集上的实验表明，LARA和B-LARA在准确率与内存效率上均优于所有基线方法。大量分析显示，该方法在从少量到多示例的不同场景下均有良好泛化能力。
原文摘要 · Abstract (English)
In-Context Learning (ICL) emerges as a key feature for Large Language Models (LLMs), allowing them to adapt to new tasks by leveraging task-specific examples without updating model parameters. However, ICL faces challenges with increasing numbers of examples due to performance degradation and quadratic computational costs. In this paper, we propose Logit Arithmetic Reweighting Approach (LARA), a novel framework that enhances ICL by using logit-based ensembling of multiple demonstrations. Our approach divides long input demonstrations into parallelizable shorter inputs to significantly reduce memory requirements, and then effectively aggregate the information by reweighting logits of each group via a non-gradient optimization approach. We further introduce Binary LARA (B-LARA), a variant that constrains weights to binary values to simplify the search space and reduces memory usage by filtering out less informative demonstration groups. Experiments on BBH and MMLU demonstrate that LARA and B-LARA outperform all baseline methods in both accuracy and memory efficiency. We also conduct extensive analysis to show that LARA generalizes well to scenarios of varying numbers of examples from limited to many-shot demonstrations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。