用视觉专家知识迁移提升视觉语言模型效率，数据量少100倍仍表现优秀。
ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts
- 通过多视觉专家动态路由知识到图像特征，提升感知能力。
- 仅用1%训练数据达到主流模型性能，推理效率显著提升。
- 适合资源有限但需高效视觉语言建模的研究者和开发者。
视觉-语言（VL）学习需要强大的视觉感知能力，如细粒度物体识别和空间感知。现有方法通常依赖在海量数据上训练大型模型来获得这些能力。本文提出一种新框架——视觉专家知识迁移（ToVE），通过预训练的视觉专家模型传递知识，实现高效的VL学习。基于冻结的CLIP编码器提供图像条件下的语言生成视觉标记，ToVE引入多个视觉专家组成的枢纽及一个标记感知门控网络，动态将专家知识路由至视觉标记。在知识迁移阶段，采用“残差知识迁移”策略，既保持视觉标记的泛化能力，又可移除贡献度低的专家以提升推理效率。进一步探索将专家知识融合至单一CLIP编码器，构建知识融合版CLIP，在部署时无需专家推理即可生成更丰富的视觉标记。跨多种VL任务的实验结果表明，所提ToVE在训练数据量减少两个数量级的情况下仍取得具有竞争力的性能。
原文摘要 · Abstract (English)
Vision-language (VL) learning requires extensive visual perception capabilities, such as fine-grained object recognition and spatial perception. Recent works typically rely on training huge models on massive datasets to develop these capabilities. As a more efficient alternative, this paper proposes a new framework that Transfers the knowledge from a hub of Vision Experts (ToVE) for efficient VL learning, leveraging pre-trained vision expert models to promote visual perception capability. Specifically, building on a frozen CLIP encoder that provides vision tokens for image-conditioned language generation, ToVE introduces a hub of multiple vision experts and a token-aware gating network that dynamically routes expert knowledge to vision tokens. In the transfer phase, we propose a "residual knowledge transfer" strategy, which not only preserves the generalizability of the vision tokens but also allows detachment of low-contributing experts to improve inference efficiency. Further, we explore to merge these expert knowledge to a single CLIP encoder, creating a knowledge-merged CLIP that produces more informative vision tokens without expert inference during deployment. Experiment results across various VL tasks demonstrate that the proposed ToVE achieves competitive performance with two orders of magnitude fewer training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。