调整提示中示例顺序，能显著提升大模型的上下文学习效果。
OptiSeq: Ordering Examples On-The-Fly for In-Context Learning
- 基于生成概率动态筛选示例顺序，无需训练即可优化推理。
- 在多个任务上提升准确率5.5至10.5个百分点。
- 适合希望不修改模型、仅优化输入顺序的开发者使用。
使用大语言模型（LLM）及其代理的应用开发者普遍反映，上下文学习（ICL）表现不稳定。本文揭示：除了示例数量与质量，其在提示中的排列顺序也会影响模型输出及性能。现有方法多依赖数据集特定技巧，而本文提出纯推理阶段、无需数据依赖的OptiSeq方法，通过分析模型生成输出的对数概率，系统性剪枝可能的排序空间，识别出高准确率与低性能的顺序模式，并推荐最优排序。在多个大模型、数据集和提示设计上的广泛实验表明，OptiSeq在各类任务中均带来5.5至10.5个百分点的准确率提升。
原文摘要 · Abstract (English)
Developers using LLMs and LLM-based agents in their applications have provided plenty of anecdotal evidence that in-context-learning (ICL) is fragile. In this paper, we show that in addition to the quantity and quality of examples, the order in which the in-context examples are listed in the prompt affects the output of the LLM and, consequently, their performance. While prior work has explored improving ICL through dataset-dependent techniques, we introduce OptiSeq, a purely inference-time, dataset-free optimization method that efficiently determines the best example order. OptiSeq leverages log probabilities of LLM-generated outputs to systematically prune the search space of possible orderings and recommend the best order(s) by distinguishing orderings that yield high levels of accuracy and those that underperform. Extensive empirical evaluation on multiple LLMs, datasets, and prompts demonstrate that OptiSeq improves accuracy by 5.5 - 10.5 percentage points across multiple tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。