动态减少视觉编码器冗余图像块，提升训练效率。
DRIP: Dynamic patch Reduction via Interpretable Pooling
- 根据输入图像自适应合并深层特征块，实现动态降维。
- 在ImageNet和CLIP上减少大量计算量，性能几乎不变。
- 适合资源有限但需高效预训练的科研团队使用。
近期，视觉语言模型在对比预训练和指令微调方面取得显著进展，推动了多模态AI的发展。然而，由于大规模预训练成本高昂，效率问题阻碍了研究者从头开始训练视觉语言模型。本文提出动态块缩减方法DRIP，通过可解释池化机制，自适应调整视觉编码器深层的特征块数量。实验表明，在从头训练ImageNet及CLIP对比预训练任务中，该方法显著降低计算量（GFLOP），同时保持相近的分类与零样本性能。为进一步验证其有效性，我们在大型生物数据集上进行了持续预训练，拓展了其在科学领域的应用潜力。
原文摘要 · Abstract (English)
Recently, the advances in vision-language models, including contrastive pretraining and instruction tuning, have greatly pushed the frontier of multimodal AI. However, owing to the large-scale and hence expensive pretraining, the efficiency concern has discouraged researchers from attempting to pretrain a vision language model from scratch. In this work, we propose Dynamic patch Reduction via Interpretable Pooling (DRIP), which adapts to the input images and dynamically merges tokens in the deeper layers of a visual encoder. Our results on both ImageNet training from scratch and CLIP contrastive pretraining demonstrate a significant GFLOP reduction while maintaining comparable classification/zero-shot performance. To further validate our proposed method, we conduct continual pretraining on a large biology dataset, extending its impact into scientific domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。