arXiv:2504.07957cs.CV2025-04ICCV被引 44

构建高质量多模态指令跟随数据集与评测基准,提升大模型精准执行指令能力。

MM-IFEngine: Towards Multimodal Instruction Following

  • 设计自动化流水线生成23,000组图像-指令对数据
  • 在多个评测集上实现最高12.3%的性能提升
  • 适合研究多模态对齐与指令理解的开发者使用

指令跟随能力衡量多模态大语言模型理解用户指令并准确执行的程度。现有训练数据稀缺,评测基准简单且评估方式不精确,难以满足严格输出约束任务的需求。为此,我们提出MM-IFEngine,一个高效的数据生成流水线,产出大规模、多样且高质量的训练数据MM-IFInstruct-23k,适用于监督微调(SFT),并扩展为用于直接偏好优化(DPO)的MM-IFDPO-23k。我们进一步构建了MM-IFEval,一个具有挑战性且多样化的多模态指令跟随评测基准,包含输出层面的组合约束和与输入图像相关的感知约束,并采用规则判断与判别模型相结合的综合评估流程。实验表明,在MM-IFInstruct-23k和MM-IFDPO-23k上微调后,模型在多个指令跟随评测集上表现显著提升,如MM-IFEval(+10.2%)、MIA(+7.6%)和IFEval(+12.3%)。相关数据集(SFT与DPO)、评测代码及训练脚本已开源至https://github.com/SYuan03/MM-IFEngine。

原文摘要 · Abstract (English)

The Instruction Following (IF) ability measures how well Multi-modal Large Language Models (MLLMs) understand exactly what users are telling them and whether they are doing it right. Existing multimodal instruction following training data is scarce, the benchmarks are simple with atomic instructions, and the evaluation strategies are imprecise for tasks demanding exact output constraints. To address this, we present MM-IFEngine, an effective pipeline to generate high-quality image-instruction pairs. Our MM-IFEngine pipeline yields large-scale, diverse, and high-quality training data MM-IFInstruct-23k, which is suitable for Supervised Fine-Tuning (SFT) and extended as MM-IFDPO-23k for Direct Preference Optimization (DPO). We further introduce MM-IFEval, a challenging and diverse multi-modal instruction-following benchmark that includes (1) both compose-level constraints for output responses and perception-level constraints tied to the input images, and (2) a comprehensive evaluation pipeline incorporating both rule-based assessment and judge model. We conduct SFT and DPO experiments and demonstrate that fine-tuning MLLMs on MM-IFInstruct-23k and MM-IFDPO-23k achieves notable gains on various IF benchmarks, such as MM-IFEval (+10.2$\%$), MIA (+7.6$\%$), and IFEval (+12.3$\%$). We have fully open-sourced the datasets (both SFT and DPO), evaluation code and training scripts at https://github.com/SYuan03/MM-IFEngine.

多模态指令跟随数据生成评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。