arXiv:2507.22887cs.CLcs.AI2025-07EMNLP被引 16

演示位置影响大模型推理效果,开头放最稳,结尾放易出错。

Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning

  • 首次发现演示位置对上下文学习有显著偏差,影响预测稳定性。
  • 演示放开头准确率最高,可提升6分;放结尾超30%预测会翻转。
  • 小模型更敏感,大模型在复杂任务上仍受轻微影响,适合提示工程优化者参考。

上下文学习(ICL)是大语言模型(LLM)的一项关键能力,通过在提示中加入少量示例(demos)实现少样本推理。然而,其性能对示例选择和顺序敏感。本文首次研究了未被探索的新型位置偏差:当示例、系统提示与用户消息在输入中的位置变化时,模型预测与准确率会发生剧烈波动。我们称此为演示位置偏差(DPP)。设计系统评估流程,在分类、问答、摘要和推理任务中验证该偏差。引入两个指标——准确率变化(ACCURACY-CHANGE)和预测变化(PREDICTION-CHANGE),量化位置变动带来的性能增益与输出波动。在来自四个开源模型族(QWEN、LLAMA3、MISTRAL、COHERE)的十种模型上进行实验,结果表明:将演示置于提示开头可获得最稳定且最高的准确率,最高提升达+6分;而将演示置于用户消息末尾时,超过30%的预测发生翻转,且问答任务正确率未提升。小模型对此敏感性更强,大模型在复杂任务上仍存在轻微影响。

原文摘要 · Abstract (English)

In-context learning (ICL) is a critical emerging capability of large language models (LLMs), enabling few-shot learning during inference by including a few demonstrations (demos) in the prompt. However, it has been found that ICL's performance can be sensitive to the choices of demos and their order. This paper investigates an unexplored new positional bias of ICL for the first time: we observe that the predictions and accuracy can drift drastically when the positions of demos, the system prompt, and the user message in LLM input are varied. We refer to this bias as DEMOS' POSITION IN PROMPT (DPP) bias. We design a systematic evaluation pipeline to study this type of positional bias across classification, question answering, summarization, and reasoning tasks. We introduce two metrics, ACCURACY-CHANGE and PREDICTION-CHANGE, to quantify net gains and output volatility induced by changes in the demos' position. Extensive experiments on ten LLMs from four open-source model families (QWEN, LLAMA3, MISTRAL, COHERE) verify that the bias significantly affects their accuracy and predictions: placing demos at the start of the prompt yields the most stable and accurate outputs with gains of up to +6 points. In contrast, placing demos at the end of the user message flips over 30\% of predictions without improving correctness on QA tasks. Smaller models are most affected by this sensitivity, though even large models remain marginally affected on complex tasks.

上下文学习提示工程位置偏差大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。