arXiv:2511.15976cs.CL2025-11被引 2

评测大模型在复杂任务对话中的指令遵循能力,构建真实场景下的高难度基准。

TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialogues

  • 将复杂指令拆解为多层级条件-动作语句,模拟真实任务流程
  • 设计三类任务,评估模型检索、纠错与按条件生成能力
  • 基于高质量人工对话语料,支持多语言和格式对比研究

在真实任务导向对话中,智能体需在多轮对话中严格遵守包含细粒度约束的复杂自然语言指令。现有基准常将指令简化为意图-槽位-API配置等简单结构,难以反映真实复杂性。为此,我们提出TOD-ProcBench,一个挑战性基准,包含从高质量ABCD数据集衍生的指令文档及对应的人工质量控制对话。我们将精细约束与操作步骤形式化为多层级条件-动作指令语句,并设计三项任务:任务1评估模型从复杂指令中检索最相关语句并预测下一步动作的能力;任务2通过注入不一致信息合成违反指令的回复,检验模型识别违规响应的能力;任务3考察模型基于原始复杂指令进行条件生成的能力。此外,我们还研究了多语言设置与不同指令文本格式对合规性能的影响。本基准已按Llama 3.3社区许可协议发布。

原文摘要 · Abstract (English)

In real-world task-oriented dialogue (TOD) settings, agents are required to strictly adhere to complex instructions while conducting multi-turn conversations with customers. These instructions are typically presented in natural language format and include general guidelines and step-by-step procedures with complex constraints. Existing TOD benchmarks often oversimplify the complex nature of these instructions by reducing them to simple schemas composed of intents, slots, and API call configurations. To address this gap and systematically benchmark LLMs' instruction-following capabilities, we propose TOD-ProcBench, a challenging benchmark featuring complex process instructions with intricate, fine-grained constraints that evaluates various LLMs' abilities to understand and follow instructions in multi-turn TODs. Our benchmark dataset comprises instruction documents derived from the high-quality ABCD dataset with corresponding conversations under human quality control. We formulate fine-grained constraints and action procedures as multi-level condition-action instruction statements. We design three tasks to comprehensively benchmark LLMs' complex instruction-following capabilities in multi-turn TODs. Task 1 evaluates how LLMs retrieve the most relevant statement from a complex instruction and predict the corresponding next action. In Task 2, we synthesize instruction-violating responses by injecting inconsistencies and manipulating the original instructions, and then we analyze how effectively LLMs can identify instruction-violating responses. Task 3 investigates LLMs' abilities in conditional generation of instruction-following responses based on the original complex instructions. Additionally, we conduct studies on the impact of multilingual settings and different instruction text formats on compliance performance. We release our benchmark under the Llama 3.3 Community License Agreement.

对话系统指令遵循大模型评测多轮对话

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。