用纯文本大模型初始化视觉编码器，让小模型也能高效完成复杂视觉任务
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
- 用纯文本LLM初始化视觉编码器，避开传统对比学习的局限
- 2B/8B小模型在文档理解等任务上超越主流大模型
- 适合移动端、机器人等资源受限场景的高效视觉语言模型
视觉语言模型（VLM）发展长期依赖模型规模扩张，限制了在手机、机器人等计算资源受限设备上的部署。本文探索紧凑型（如2B和8B）VLM的性能极限，挑战主流观点——顶尖VLM必须依赖大规模对比预训练的视觉编码器（如CLIP/SigLIP）。我们发现对比学习优化于判别性，强制粗粒度类别不变性，抑制了密集描述和复杂推理所需的细粒度视觉线索。为此，我们提出Penguin-VL，其视觉编码器由纯文本LLM初始化。实验表明，Penguin-Encoder优于传统对比预训练编码器，显著提升视觉保真度与数据效率。在多种图像和视频基准上，Penguin-VL在数学推理上媲美领先模型（如Qwen3-VL），并在文档理解、视觉知识、多视角视频理解等任务中表现更优。这些成果在轻量架构下实现，证明视觉表征提升比模型缩放是性能增长的关键驱动力。消融实验显示，Penguin-Encoder始终优于对比预训练编码器，有效保留对密集感知和复杂推理至关重要的细粒度空间与时间线索。该方案为计算高效VLM提供强兼容替代，适用于资源受限场景。
原文摘要 · Abstract (English)
Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。