发现大模型对多约束指令顺序敏感，先难后易更高效
Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following
- 设计难度分布指数量化约束顺序影响
- 先难后易顺序下模型表现显著提升
- 适用于不同架构与规模的模型，适合提示工程优化
现实世界中的多约束指令对现有大语言模型构成重大挑战。观察发现，当约束条件顺序被打乱时，模型性能会出现剧烈波动。然而，现有研究尚未系统探讨多约束指令遵循中的位置偏差问题。为此，我们设计了一项探测任务，通过新颖的约束难度分布指数（CDDI）定量衡量约束难度分布。实验结果表明，当约束按‘难到易’顺序呈现时，模型表现更优，该偏好可推广至不同架构和参数规模的模型。此外，我们开展了解释性研究，直观揭示了模型注意力与约束顺序之间的关联。代码与数据集已公开于 https://github.com/meowpass/PBIF。
原文摘要 · Abstract (English)
Real-world instructions with multiple constraints pose a significant challenge to existing large language models (LLMs). An observation is that the LLMs exhibit dramatic performance fluctuation when disturbing the order of the incorporated constraints. Yet, none of the existing works has systematically investigated this position bias problem in the field of multi-constraint instruction following. To bridge this gap, we design a probing task where we quantitatively measure the difficulty distribution of the constraints by a novel Difficulty Distribution Index (CDDI). Through the experimental results, we find that LLMs are more performant when presented with the constraints in a ``hard-to-easy'' order. This preference can be generalized to LLMs with different architecture or different sizes of parameters. Additionally, we conduct an explanation study, providing an intuitive insight into the correlation between the LLM's attention and constraint orders. Our code and dataset are publicly available at https://github.com/meowpass/PBIF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。