测试大模型同时执行500条指令的能力，发现最佳模型准确率仅68%。
How Many Instructions Can LLMs Follow at Once?
- 构建包含500条关键词指令的基准测试IFScale，模拟高密度场景
- 20个主流模型在500条指令下最高准确率68%，性能随指令数下降明显
- 揭示模型大小、推理能力与指令偏倚等关键影响因素，适合提示工程研究者
生产级大模型系统需同时遵循数十甚至上百条指令，但现有评测仅针对单条或少量指令，无法反映高密度指令下的表现。本文提出IFScale基准，包含500条用于撰写商业报告的关键词包含指令，用于衡量指令密度增加时模型遵循能力的退化情况。我们在7家主要厂商的20个前沿模型上进行评估，发现即使最优模型在最大密度（500条）下准确率也仅为68%。分析显示，模型规模和推理能力与三种不同的性能退化模式相关，存在对早期指令的偏好，以及多种指令遵循错误类型。研究结果可指导真实应用中高密度提示的设计，并揭示性能与延迟的重要权衡。所有数据与代码已开源：https://distylai.github.io/IFScale。
原文摘要 · Abstract (English)
Production-grade LLM systems require robust adherence to dozens or even hundreds of instructions simultaneously. However, the instruction-following capabilities of LLMs at high instruction densities have not yet been characterized, as existing benchmarks only evaluate models on tasks with a single or few instructions. We introduce IFScale, a simple benchmark of 500 keyword-inclusion instructions for a business report writing task to measure how instruction-following performance degrades as instruction density increases. We evaluate 20 state-of-the-art models across seven major providers and find that even the best frontier models only achieve 68% accuracy at the max density of 500 instructions. Our analysis reveals model size and reasoning capability to correlate with 3 distinct performance degradation patterns, bias towards earlier instructions, and distinct categories of instruction-following errors. Our insights can help inform design of instruction-dense prompts in real-world applications and highlight important performance-latency tradeoffs. We open-source the benchmark and all results for further analysis at https://distylai.github.io/IFScale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。