通过元分析发现：冻结大部分类别模型可大幅节省算力，且不影响下游性能。
Renaissance: Investigating the Pretraining of Vision-Language Encoders
- 采用元分析方法系统评估视觉语言模型预训练策略
- 冻结大部分模型参数可降低算力消耗，性能几乎不变
- 框架支持灵活构建与评估，适合研究者快速实验
近年来视觉语言（VL）任务模型迅速发展，但其设计与训练的最佳实践仍不明确。现有编程工具有限，增加了研究难度。本文通过元分析回答多个关于视觉语言编码器预训练的问题，并提出名为Renaissance的评估框架。实验表明，在预训练阶段冻结模型大部分参数，可在几乎不损失下游任务性能的前提下显著节省计算资源。另一组实验对比了基于视觉模型或文本模型构建VL Transformer的效果。Renaissance提供了创建、训练和评估变压器编码器的高度灵活性，源代码将在发表后公开，地址为https://github.com/bsu-slim/renaissance。
原文摘要 · Abstract (English)
In the past several years there has been an explosion of available models for vision-language (VL) tasks. Unfortunately, the literature still leaves open a number of questions related to best practices in designing and training such models. Additionally, the limited programming tools available for modeling make conducting VL research more difficult than necessary. In this paper, we seek to answer several questions related to the pretraining of VL encoders through meta-analysis. To conduct these experiments, we introduce a VL evaluation framework called Renaissance. In our first set of experiments, we show that we can save significant compute at little to no cost to downstream performance, by freezing large parts of VL models during pretraining. In our second set of experiments, we examine the effect of basing a VL transformer on a vision model versus a text model. Renaissance offers a great deal of flexibility in creating, training and evaluating transformer encoders for VL modeling. Its source code will be made publicly available upon publication. The source code for Renaissance can be found at https://github.com/bsu-slim/renaissance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。