发现提示顺序影响模型答题，根源是注意力机制限制信息流动。
Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models
- 用因果注意力分析提示结构对模型的影响。
- 上下文在前时准确率比在后高14个百分点以上。
- 适合研究提示工程与模型注意力机制的读者。
大型语言模型对提示结构表现出意外敏感性，但其内在机制尚不明确。本文深入研究了一个显著现象：在多项选择题问答中，将上下文置于问题和选项之前（CQO）的格式，相比反之（QOC）能提升超过14个百分点的准确率，且在多种模型和数据集上均保持一致。通过系统的架构分析，我们识别出因果注意力是核心机制：在QOC提示中，因果掩码阻止选项令牌关注上下文，造成信息瓶颈，使上下文对选项不可见。
原文摘要 · Abstract (English)
Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets. Through systematic architectural analysis, we identify causal attention as the core mechanism: in QOC prompts, the causal mask prevents option tokens from attending to context, creating an information bottleneck where context becomes invisible to options.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。