针对大模型推理缺陷生成针对性合成数据,提升性能。
Learning from Reasoning Failures via Synthetic Data Generation
- 分析弱模型错误,用强模型生成修复缺陷的合成数据
- 构建超55万条数据集,显著提升多任务表现
- 适合需要改进推理能力的视觉语言模型研究者
在生成式AI训练中,合成数据策略日益重要,尤其对多模态大模型(LMMs)而言,因高质量图文配对数据稀缺,该方法尤为关键。现有合成数据生成方法未针对LMM推理能力的具体缺陷进行优化。受人类通过关注自身失败领域来高效学习的启发,我们提出一种新方法:利用前沿模型自动分析弱LMM的推理错误,并生成可修复这些缺陷的合成示例,再经筛选确保质量。基于此方法,我们构建了一个包含超过553,000个样本的大型多模态指令微调数据集,并通过大量实验验证其有效性。结果表明,使用该合成数据训练的模型,在多个下游任务上的表现甚至超越使用同等数量真实数据训练的模型,证明了针对特定推理缺陷生成合成数据的高价值。相关数据集与代码将公开发布。
原文摘要 · Abstract (English)
Training models on synthetic data has emerged as an increasingly important strategy for improving the performance of generative AI. This approach is particularly helpful for large multimodal models (LMMs) due to the relative scarcity of high-quality paired image-text data compared to language-only data. While a variety of methods have been proposed for generating large multimodal datasets, they do not tailor the synthetic data to address specific deficiencies in the reasoning abilities of LMMs which will be trained with the generated dataset. In contrast, humans often learn in a more efficient manner by seeking out examples related to the types of reasoning where they have failed previously. Inspired by this observation, we propose a new approach for synthetic data generation which is grounded in the analysis of an existing LMM's reasoning failures. Our methodology leverages frontier models to automatically analyze errors produced by a weaker LMM and propose new examples which can be used to correct the reasoning failure via additional training, which are then further filtered to ensure high quality. We generate a large multimodal instruction tuning dataset containing over 553k examples using our approach and conduct extensive experiments demonstrating its utility for improving the performance of LMMs on multiple downstream tasks. Our results show that models trained on our synthetic data can even exceed the performance of LMMs trained on an equivalent amount of additional real data, demonstrating the high value of generating synthetic data targeted to specific reasoning failure modes in LMMs. We will make our dataset and code publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。