arXiv:2409.04828cs.CVcs.AI2024-09被引 18

用轻量策略提升视觉语言模型性能,效果媲美顶尖模型。

POINTS: Improving Your Vision-language Model with Affordable Strategies

  • 通过系统性消融实验优化训练策略,提升模型鲁棒性。
  • 用困惑度筛选数据,构建100万条高质量预训练集。
  • 采用模型汤技术融合多数据集,避免增量收益递减。

近年来,视觉语言模型在光学字符识别和几何问题求解等任务上取得显著进展。然而仍存在三大问题:1)商用模型架构不透明,开源模型缺乏训练策略的详细分析;2)开源数据集选择多凭经验,预训练数据未充分探索;3)微调过度依赖增加数据集,导致收益递减。为此,本文提出三项改进:1)基于最新技术训练稳健基线模型,对每项改进进行系统消融与验证;2)受大语言模型启发,使用困惑度过滤预训练数据,筛选出最低困惑度样本构建100万条数据集,实现优异性能;3)在视觉指令微调阶段,当新增数据集带来的提升趋缓时,采用模型汤方法融合不同数据集。最终得到一个90亿参数的模型,表现媲美当前最优模型。这些策略高效且轻量,便于社区广泛采用。

原文摘要 · Abstract (English)

In recent years, vision-language models have made significant strides, excelling in tasks like optical character recognition and geometric problem-solving. However, several critical issues remain: 1) Proprietary models often lack transparency about their architectures, while open-source models need more detailed ablations of their training strategies. 2) Pre-training data in open-source works is under-explored, with datasets added empirically, making the process cumbersome. 3) Fine-tuning often focuses on adding datasets, leading to diminishing returns. To address these issues, we propose the following contributions: 1) We trained a robust baseline model using the latest advancements in vision-language models, introducing effective improvements and conducting comprehensive ablation and validation for each technique. 2) Inspired by recent work on large language models, we filtered pre-training data using perplexity, selecting the lowest perplexity data for training. This approach allowed us to train on a curated 1M dataset, achieving competitive performance. 3) During visual instruction tuning, we used model soup on different datasets when adding more datasets yielded marginal improvements. These innovations resulted in a 9B parameter model that performs competitively with state-of-the-art models. Our strategies are efficient and lightweight, making them easily adoptable by the community.

视觉语言模型优化数据筛选模型汤

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。