顺序比选例更重要：同组例子换顺序，性能波动堪比换一组
Order Matters: Rethinking Prompt Construction in In-Context Learning
- 通过控制实验发现，例子排序对模型表现影响与选例相当
- 不同顺序导致性能差异可达与更换例子集同等水平
- 仅用开发集就能找到接近最优的顺序，适合优化提示工程
上下文学习（ICL）使大语言模型通过示例序列完成新任务。以往研究普遍认为示例选择比顺序更重要，因此聚焦于选例。我们重新审视这一假设，在分类与生成任务上，使用多个开源模型（0.5B至27B参数）及GPT-5进行系统对比，发现不同示例顺序带来的性能波动，与使用完全不同的示例集相当。此外，仅利用开发集即可识别出表现优异的顺序，其性能接近基于测试标签的最优顺序。结果表明，示例选择与排序在提示设计中具有同等重要性且相互交织，需重新审视现有假设。
原文摘要 · Abstract (English)
In-context learning (ICL) enables large language models to perform new tasks by conditioning on a sequence of examples. Most prior work reasonably and intuitively assumes that which examples are chosen has a far greater effect on performance than how those examples are ordered, leading to a focus on example selection. We revisit this assumption and conduct a systematic comparison between the effect of selection and ordering. Through controlled experiments on both classification and generation tasks, using multiple open-source model families (0.5B to 27B parameters) and GPT-5, we find that the variance in performance due to different example orderings is comparable to that from using entirely different example sets. Furthermore, we show that strong orderings can be identified using only a development set, achieving performance close to an oracle that selects the best ordering based on test labels. Our findings highlight the equal and intertwined importance of example selection and ordering in prompt design, calling for a reexamination of the assumptions held in ICL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。