构建首个百万级多模态智能体推理数据集,提升模型逻辑推理与工具使用能力。
MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
- 用四阶段流程生成包含推理与反思的多轮对话数据
- 在8个基准上平均提升2.7%,RAG任务提升8.8%
- 适合研究多模态推理、智能体系统与工具调用的开发者
大型语言模型通过智能体微调,在思维链(CoT)和工具使用方面表现出色,显著优于独立模型。然而,多模态领域仍缺乏大规模高质量的智能体训练数据,制约了多模态大模型潜力的发挥。为此,我们提出MMAT-1M,首个百万级多模态智能体微调数据集,支持思维链、反思与动态工具调用。数据集通过四阶段构建:1)收集公开多模态问答数据;2)利用GPT-4o生成推理过程,并在多轮对话中动态集成API调用与检索增强生成(RAG)信息;3)通过反思机制优化推理内容,确保逻辑一致性和准确性,形成包含推理与反思(RR)的多轮对话数据;4)为提升效率,可将多轮对话压缩为单轮推理与反思(ORR)格式。在开源多模态模型上微调后,如InternVL2.5-8B-RR,在8个公开基准上平均提升2.7%,在RAG基准Dyn-VQA上提升8.8%,验证了该数据集在增强多模态推理与工具调用能力方面的有效性。数据集已公开于https://github.com/VIS-MPU-Agent/MMAT-1M。
原文摘要 · Abstract (English)
Large Language Models (LLMs), enhanced through agent tuning, have demonstrated remarkable capabilities in Chain-of-Thought (CoT) and tool utilization, significantly surpassing the performance of standalone models. However, the multimodal domain still lacks a large-scale, high-quality agent tuning dataset to unlock the full potential of multimodal large language models. To bridge this gap, we introduce MMAT-1M, the first million-scale multimodal agent tuning dataset designed to support CoT, reflection, and dynamic tool usage. Our dataset is constructed through a novel four-stage data engine: 1) We first curate publicly available multimodal datasets containing question-answer pairs; 2) Then, leveraging GPT-4o, we generate rationales for the original question-answer pairs and dynamically integrate API calls and Retrieval Augmented Generation (RAG) information through a multi-turn paradigm; 3) Furthermore, we refine the rationales through reflection to ensure logical consistency and accuracy, creating a multi-turn dialogue dataset with both Rationale and Reflection (RR); 4) Finally, to enhance efficiency, we optionally compress multi-turn dialogues into a One-turn Rationale and Reflection (ORR) format. By fine-tuning open-source multimodal models on the MMAT-1M, we observe significant performance gains. For instance, the InternVL2.5-8B-RR model achieves an average improvement of 2.7% across eight public benchmarks and 8.8% on the RAG benchmark Dyn-VQA, demonstrating the dataset's effectiveness in enhancing multimodal reasoning and tool-based capabilities. The dataset is publicly available at https://github.com/VIS-MPU-Agent/MMAT-1M.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。