构建多维指令遵循评估框架，提升大模型精准执行复杂指令能力
MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models
- 设计三类约束模式、四类类别与四级难度的评估体系
- 18个模型在四级难度下准确率从80.82%降至36.76%
- 生成9106条可验证数据，训练后模型指令遵循能力显著提升
指令遵循指大语言模型生成满足所有约束条件输出的能力。现有研究多聚焦于约束类别，评估维度有限，难以指导模型优化。为此，我们提出MulDimIF，一个包含三类约束模式、四类约束类别和四级难度的多维评估框架。基于此框架，设计可控指令生成流程，通过约束扩展、冲突检测与指令重写，构建了9106条可代码验证样本。评估18个来自六个模型家族的LLM发现，性能在不同约束设置下差异显著：平均准确率从第I级的80.82%下降至第IV级的36.76%。此外，使用本框架生成的数据进行训练，能有效提升指令遵循能力，且不损害通用性能。深入分析表明，这种提升主要源于注意力模块参数更新，增强了模型对约束的识别与遵守能力。代码与数据见https://github.com/Junjie-Ye/MulDimIF。
原文摘要 · Abstract (English)
Instruction following refers to the ability of large language models (LLMs) to generate outputs that satisfy all specified constraints. Existing research has primarily focused on constraint categories, offering limited evaluation dimensions and little guidance for improving instruction-following abilities. To address this gap, we introduce MulDimIF, a multi-dimensional constraint framework encompassing three constraint patterns, four constraint categories, and four difficulty levels. Based on this framework, we design a controllable instruction generation pipeline. Through constraint expansion, conflict detection, and instruction rewriting, we construct 9,106 code-verifiable samples. We evaluate 18 LLMs from six model families and find marked performance differences across constraint settings. For instance, average accuracy decreases from 80.82% at Level I to 36.76% at Level IV. Moreover, training with data generated by our framework significantly improves instruction following without compromising general performance. In-depth analysis indicates that these gains stem largely from parameter updates in attention modules, which strengthen constraint recognition and adherence. Code and data are available in https://github.com/Junjie-Ye/MulDimIF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。