测试大模型对复杂逻辑指令的理解能力,发现多数模型正确率不足60%。
LogicIF: Towards Complex Logic Instruction Following
- 构建自动化框架生成可验证的逻辑指令,涵盖条件、循环等复杂结构。
- 创建426条人工审核的逻辑密集型指令评测集,揭示当前模型严重不足。
- 提供2.7万条合成训练数据,强化学习训练后模型性能显著提升。
指令遵循推动了大语言模型的发展,是推理与自主行为的基础。随着任务复杂度上升,自然语言中嵌入的逻辑结构日益复杂,但大模型在处理此类指令上的表现仍缺乏研究。本文提出逻辑指令遵循(LogicIF)任务,要求模型精确执行复杂指令中的每一步逻辑。为此,我们构建了可扩展的自动化生成框架LogicIFGen,从代码函数生成可验证的逻辑指令,自然表达条件、循环、嵌套和函数调用等结构。基于此,我们整理出426条经人工审核的逻辑丰富指令构成LogicIFEval基准。实验表明,当前最先进大模型在该基准上正确率普遍低于60%,暴露出处理复杂逻辑指令的重大缺陷。为验证框架有效性,我们收集广泛难度的代码函数,生成包含27,324条可验证指令的LogicIFTrain训练集。使用强化学习训练的模型在领域内与跨领域基准上均显著优于基线模型。所有数据、代码与模型已开源。
原文摘要 · Abstract (English)
Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors. As tasks grow more challenging, the logic structures embedded in natural language instructions becomes increasingly intricate. However, how well LLMs perform on such logic-intensive instructions remains under-explored. We propose Logic Instruction Following (LogicIF), a task requiring models to precisely follow and simulate every logic step within complex instructions. To generate these instructions, we construct LogicIFGen, a scalable, automated framework for generating verifiable instructions from code functions, which can naturally express rich logic such as conditions, loops, nesting, and function calls. Using this framework, we curate a collection of complex code functions to construct LogicIFEval, a benchmark comprising 426 verifiable, human-examined, logic-rich instructions. Our experiments demonstrate that current state-of-the-art LLMs still struggle to correctly follow the instructions in LogicIFEval. Most LLMs can only follow fewer than 60% of the instructions, revealing significant deficiencies in their capacity to handle instructions that involve complex logic structures. To further demonstrate the efficacy of our framework, we collect an extensive set of code functions across a wide range of logic difficulties to generate LogicIFTrain, a fully synthesized training set featuring 27,324 verifiable instructions. Our results show that a model trained on LogicIFTrain using reinforcement learning significantly outperforms the base model on both in-domain and out-of-domain benchmarks. All the data, codes and models are released at https://github.com/mianzhang/LogicIF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。