构建多轮指令跟随的结构化评估基准，揭示大模型对话结构理解短板。
StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following
- 提出六类跨轮次关系，建模对话结构依赖性。
- 13个主流模型普遍存在多轮结构理解不足。
- 适合研究对话系统与大模型评估的学者使用。
多轮指令遵循能力是大语言模型在实际应用中的核心竞争力。现有评估基准主要关注细粒度约束满足和领域特定能力，却忽略了区分多轮与单轮交互的关键对话轮次间结构依赖。这些结构依赖不仅反映用户意图，更在约束满足之外构建了指令遵循评估的第二维。为此，我们提出 StructFlowBench，一个具备结构流建模的多轮指令遵循基准。该基准定义了六种基本的跨轮关系，引入新型结构约束用于模型评估，并可作为生成参数定制特定场景对话流。采用成熟的基于LLM的自动评估方法，对13个领先开源与闭源大模型进行系统评估。实验结果揭示当前模型在多轮对话结构理解上存在显著缺陷。代码已公开于 https://github.com/MLGroupJLU/StructFlowBench。
原文摘要 · Abstract (English)
Multi-turn instruction following capability constitutes a core competency of large language models (LLMs) in real-world applications. Existing evaluation benchmarks predominantly focus on fine-grained constraint satisfaction and domain-specific capability assessment, yet overlook the crucial structural dependencies between dialogue turns that distinguish multi-turn from single-turn interactions. These structural dependencies not only reflect user intent but also establish an essential second dimension for the instruction following evaluation beyond constraint satisfaction. To address this gap, we propose StructFlowBench, a multi-turn instruction following benchmark with structural flow modeling. The benchmark defines an innovative structural flow framework with six fundamental inter-turn relationships. These relationships introduce novel structural constraints for model evaluation and also serve as generation parameters for creating customized dialogue flows tailored to specific scenarios. Adopting established LLM-based automatic evaluation methodologies, we conduct systematic evaluations of 13 leading open-source and closed-source LLMs. Experimental results reveal significant deficiencies in current models' comprehension of multi-turn dialogue structures. The code is available at https://github.com/MLGroupJLU/StructFlowBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。