arXiv:2509.21051cs.CL2025-09EMNLP被引 5

测试大模型同时执行多个指令的能力,发现越多越差,但少量数据可准确预测。

When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following

  • 设计两个新评测集,分别测文本和代码生成中多指令跟随能力。
  • 模型性能随指令数增加而下降,10条指令时明显变差。
  • 只需几百样本即可用回归模型预测未知组合表现,适合快速评估。

随着大语言模型在真实场景中的广泛应用,理解其同时遵循多个指令的能力变得至关重要。为此,我们构建了两个专用评测基准:用于文本生成的Many Instruction-Following Eval(ManyIFEval),支持最多十条指令;以及用于代码生成的Style-aware Mostly Basic Programming Problems(StyleMBPP),支持最多六条指令。在十个LLM上进行实验发现,随着指令数量增加,性能持续下降。由于实际应用中所有指令组合的评估计算成本过高,我们开发了三种回归模型,可估计未见指令组合及训练中未覆盖的指令数量下的表现。结果显示,仅以指令数量为变量的逻辑回归模型,即可在未见组合上实现约10%误差的预测。我们还发现,仅需500个样本(ManyIFEval)或300个样本(StyleMBPP)即可有效完成性能估计,显著提升评估效率。

原文摘要 · Abstract (English)

As large language models (LLMs) are increasingly applied to real-world scenarios, it becomes crucial to understand their ability to follow multiple instructions simultaneously. To systematically evaluate these capabilities, we introduce two specialized benchmarks for fundamental domains where multiple instructions following is important: Many Instruction-Following Eval (ManyIFEval) for text generation with up to ten instructions, and Style-aware Mostly Basic Programming Problems (StyleMBPP) for code generation with up to six instructions. Our experiments with the created benchmarks across ten LLMs reveal that performance consistently degrades as the number of instructions increases. Furthermore, given the fact that evaluating all the possible combinations of multiple instructions is computationally impractical in actual use cases, we developed three types of regression models that can estimate performance on both unseen instruction combinations and different numbers of instructions which are not used during training. We demonstrate that a logistic regression model using instruction count as an explanatory variable can predict performance of following multiple instructions with approximately 10% error, even for unseen instruction combinations. We show that relatively modest sample sizes (500 for ManyIFEval and 300 for StyleMBPP) are sufficient for performance estimation, enabling efficient evaluation of LLMs under various instruction combinations.

多指令跟随模型评估性能预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。