测试大模型在乱序指令下的表现,发现其严重依赖顺序结构。
RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures
- 用打乱顺序的指令测试模型,分离结构与内容影响
- 乱序时准确率最高下降72%,显示对顺序高度敏感
- 适合研究模型指令遵循能力或自动化系统设计者
大型语言模型(LLMs)被广泛用于复杂工作流,但其保持指令连贯性的能力尚未充分探索。现有基准将任务复杂度与结构顺序混杂,难以分离提示拓扑的影响。我们提出RIFT(Reordered Instruction Following Testbed),通过重排《危险边缘》问答对,测试模型在两种提示结构下的表现:线性提示(按序执行)与跳跃提示(内容相同但需非顺序遍历)。在六种领先开源LLM上进行10,000次评估,跳跃条件下准确率最高下降72%(相比基线),揭示模型对位置连续性存在强依赖。错误分析显示约50%失败源于指令顺序违规和语义漂移,表明当前架构将指令遵循视为序列模式而非推理能力。这一结构性敏感性是当前模型的根本局限,对需要非顺序控制流的应用(如工作流自动化、多智能体系统)具有直接影响。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly relied upon for complex workflows, yet their ability to maintain flow of instructions remains underexplored. Existing benchmarks conflate task complexity with structural ordering, making it difficult to isolate the impact of prompt topology on performance. We introduce RIFT, Reordered Instruction Following Testbed, to assess instruction following by disentangling structure from content. Using rephrased Jeopardy! question-answer pairs, we test LLMs across two prompt structures: linear prompts, which progress sequentially, and jumping prompts, which preserve identical content but require non-sequential traversal. Across 10,000 evaluations spanning six state-of-the-art open-source LLMs, accuracy dropped by up to 72% under jumping conditions (compared to baseline), revealing a strong dependence on positional continuity. Error analysis shows that approximately 50% of failures stem from instruction-order violations and semantic drift, indicating that current architectures internalize instruction following as a sequential pattern rather than a reasoning skill. These results reveal structural sensitivity as a fundamental limitation in current architectures, with direct implications for applications requiring non-sequential control flow such as workflow automation and multi-agent systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。