研究大模型对输入顺序的敏感性，发现顺序变化会影响输出准确性。
The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs
- 通过多任务实验测试输入顺序对模型的影响
- 打乱输入顺序导致准确率明显下降
- 现有提示技巧无法完全解决此问题，适合高风险场景使用者关注
随着大语言模型（LLMs）在各类应用中日益重要，确保其在不同输入条件下的可靠性至关重要。其中一个关键问题是顺序敏感性：输入排列的微小变化可能导致输出不一致或存在偏差。尽管近期进展已部分降低该敏感性，但问题仍未解决。本文针对用户无法访问内部结构的模型（如闭源模型或通过API调用的模型），在多个任务中开展实验，包括改写、相关性判断和选择题。结果表明，输入顺序显著影响各任务表现，打乱输入会导致输出准确率明显下降。少样本提示虽有一定缓解作用，但效果有限且无法彻底解决问题。研究揭示了在高风险应用中的持续风险，强调未来需开发更鲁棒的模型或改进输入处理技术。
原文摘要 · Abstract (English)
As large language models (LLMs) become integral to diverse applications, ensuring their reliability under varying input conditions is crucial. One key issue affecting this reliability is order sensitivity, wherein slight variations in the input arrangement can lead to inconsistent or biased outputs. Although recent advances have reduced this sensitivity, the problem remains unresolved. This paper investigates the extent of order sensitivity in LLMs whose internal components are hidden from users (such as closed-source models or those accessed via API calls). We conduct experiments across multiple tasks, including paraphrasing, relevance judgment, and multiple-choice questions. Our results show that input order significantly affects performance across tasks, with shuffled inputs leading to measurable declines in output accuracy. Few-shot prompting demonstrates mixed effectiveness and offers partial mitigation; however, fails to fully resolve the problem. These findings highlight persistent risks, particularly in high-stakes applications, and point to the need for more robust LLMs or improved input-handling techniques in future development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。