去除非必要模块，用纯生成式训练提升视觉编码器效率
OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning
- 仅保留图像与文本生成任务，移除文本编码器和对比损失
- 训练时间减半、显存降低1.8倍，最大批量扩大至8000
- 适合追求高效训练的多模态模型研发人员
本文对OpenVision架构与损失设计进行简化，以提升训练效率。受先前视觉语言预训练方法CapPa和AIMv2，以及现代多模态设计如LLaVA启发，我们移除了文本编码器（及其对应的对比损失），仅保留生成式描述损失作为训练信号，形成新版本OpenVision 2。初步结果表明，尽管结构简化，OpenVision 2在多种多模态基准测试中性能与原模型相当，同时显著降低训练时间和显存消耗：以ViT-L/14为例，训练时间从83小时降至57小时（约1.5倍加速），显存使用从24.5GB降至13.8GB（约1.8倍压缩），最大批量规模从2000提升至8000。该高效训练能力使模型规模突破原限制，达超过10亿参数。我们认为，这种轻量级、纯生成式范式对未来的多模态基础模型视觉编码器发展极具潜力。
原文摘要 · Abstract (English)
This paper provides a simplification on OpenVision's architecture and loss design for enhancing its training efficiency. Following the prior vision-language pretraining works CapPa and AIMv2, as well as modern multimodal designs like LLaVA, our changes are straightforward: we remove the text encoder (and therefore the contrastive loss), retaining only the captioning loss as a purely generative training signal. We name this new version OpenVision 2. The initial results are promising: despite this simplification, OpenVision 2 competitively matches the original model's performance on a broad set of multimodal benchmarks while substantially cutting both training time and memory consumption. For example, with ViT-L/14, it reduces training time by about 1.5x (from 83h to 57h), and memory usage by about 1.8x (from 24.5GB to 13.8GB, equivalently allowing the maximum batch size to grow from 2k to 8k). This superior training efficiency also allows us to scale far beyond the largest vision encoder used in OpenVision, reaching more than 1 billion parameters. We hold a strong belief that this lightweight, generative-only paradigm is compelling for future vision encoder development in multimodal foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。