研究指令模板规模对多模态模型训练的影响，发现中等规模效果最佳。
Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model
- 用程序化生成超1.5万种不同指令模板，系统测试其训练影响。
- 模型性能随模板数量增加先升后降，中等规模时达到峰值。
- 最优模板训练可提升10%性能，优于同类模型75倍数据增强结果。
当前多模态语言模型（MLM）训练方法忽略了指令模板的影响。以往研究采用手工或模型生成的模板，未能系统探究模板规模对训练的效果。本文提出一种程序化指令模板生成器，通过在加权采样的元模板中填入随机位置同义词，生成超过15,000种独特指令模板，从而全面探索不同模板规模对MLM训练性能的影响。实验表明，模型能力并非随模板规模增大而持续提升，最优性能出现在中等模板规模。在最优模板规模下进行数据增强训练的模型，相比原始数据训练的模型性能提升最高达10%，且整体表现优于仅使用最多75倍原始数据量增强的同类模型。代码将公开于 https://github.com/shijian2001/TemplateScaling。
原文摘要 · Abstract (English)
Current multimodal language model (MLM) training approaches overlook the influence of instruction templates. Previous research deals with this problem by leveraging hand-crafted or model-generated templates, failing to investigate the scaling effect of instruction templates on MLM training. In this work, we propose a programmatic instruction template generator capable of producing over 15K unique instruction templates by filling randomly sampled positional synonyms into weighted sampled meta templates, enabling us to comprehensively explore MLM's performance across various template scales in the training process. Our investigation into scaling instruction templates for MLM training demonstrates that MLM capabilities do not consistently improve with increasing template scale. Instead, optimal performance is achieved at a medium template scale. Models trained with data augmented at the optimal template scale achieve performance gains of up to 10% over those trained on the original data and achieve the best overall performance compared with the similar-scale MLMs tuned on at most 75 times the scale of our augmented dataset. The code will be publicly available at https://github.com/shijian2001/TemplateScaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。