测试大模型对嵌套指令的遵循能力，发现现有模型表现不佳。
IFHierBench: Hierarchical Instruction Following for Large Language Models
- 构建分层指令基准，支持多级约束检查。
- 最强模型仅50%准确率，深度越深性能越差。
- 适合研究模型细粒度指令理解与训练方法的人。
指令遵循能力对大模型在真实场景中的部署至关重要，下游组件依赖输出满足特定约束。当前部署常通过单次LLM调用完成任务，一个提示需生成具有层级结构的输出，整体、结构段落及嵌套字段均需满足具体约束。现有基准将约束视为扁平列表，无法针对输出特定部分进行校验。我们提出IFHierBench，包含600个分层提示，覆盖四种约束树深度和35种不同约束，每个提示配有确定性检查器，在每个作用域验证约束满足情况。评估七个主流专有及开源模型发现，即使最强模型也仅略超50%的提示级准确率，且随着约束深度增加，准确率急剧下降。当前大模型在可靠遵循嵌套约束方面仍存在显著差距，亟需未来训练方法关注更细粒度的约束遵循以提升指令遵循能力。
原文摘要 · Abstract (English)
Instruction-following ability is critical for deploying large language models in real-world applications, where downstream components depend on the output satisfying specific constraints. Modern deployments increasingly handle the full task in a single LLM call, with one prompt specifying a layered output whose overall artifact, structural sections, and nested fields must each satisfy concrete constraints. Existing instruction-following benchmarks treat the constraint set as a flat list applied uniformly to the response, so they cannot scope a check to a particular section of the output. We introduce IFHierBench, a hierarchical instruction-following benchmark of 600 prompts stratified across four constraint-tree depths and 35 distinct constraints, each prompt paired with a deterministic checker that verifies satisfaction at every scope. Evaluating seven leading proprietary and open-weight models, we find that even the strongest model only marginally exceeds 50% prompt-level accuracy and that accuracy degrades sharply as constraint depth grows. Reliably following nested constraints remains a substantial gap for current LLMs, motivating future training methods that consider constraint adherence at finer granularity to achieve better instruction-following ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。