发现语言模型越强，越能理解视觉概念，还能少用数据训练出好视觉模型。
Better Language Models Exhibit Higher Visual Alignment
- 用冻结语言模型测试视觉对齐能力，发现解码器优于编码器。
- 仅用563k图文对和不到1小时算力，图像分类准确率达51%。
- 跨语言场景下显著超越CLIP，中文图像分类准确率38.7%。
文本仅有的大语言模型（LLMs）与视觉世界对齐程度如何？我们通过将各类语言模型的冻结表征融入判别式视觉-语言框架，评估其零样本泛化至新概念的能力。结果表明，即使控制模型和数据集规模，解码器模型的视觉对齐能力仍强于编码器。语言建模性能与视觉泛化能力正相关，说明单模态语言模型的进步可同步提升视觉模型表现。基于此，我们提出ShareLock，一种轻量级融合冻结视觉与语言主干的方法。ShareLock在多种任务上表现稳健，大幅减少配对数据和计算需求：仅需563,000张图像-标题对，训练时间不足一GPU小时，即在ImageNet上达到51%准确率；在跨语言场景中，其中国语图像分类准确率达到38.7%，远超CLIP的1.4%。代码已公开。
原文摘要 · Abstract (English)
How well do text-only large language models (LLMs) align with the visual world? We present a systematic evaluation of this question by incorporating frozen representations of various language models into a discriminative vision-language framework and measuring zero-shot generalization to novel concepts. We find that decoder-based models exhibit stronger visual alignment than encoders, even when controlling for model and dataset size. Moreover, language modeling performance correlates with visual generalization, suggesting that advances in unimodal LLMs can simultaneously improve vision models. Leveraging these insights, we propose ShareLock, a lightweight method for fusing frozen vision and language backbones. ShareLock achieves robust performance across tasks while drastically reducing the need for paired data and compute. With just 563k image-caption pairs and under one GPU-hour of training, it reaches 51% accuracy on ImageNet. In cross-lingual settings, ShareLock dramatically outperforms CLIP, achieving 38.7% top-1 accuracy on Chinese image classification versus CLIP's 1.4%. Code is available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。