用高质量局部描述训练视觉模型,提升细粒度感知能力。
FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions
- 用密集重描述数据逐步训练高分辨率视觉编码器
- 在长上下文检索任务中达到顶尖零样本性能
- 适合需要精细视觉理解的多模态大模型研究者
尽管多模态大语言模型发展迅速,其视觉编码器常成为性能瓶颈。传统基于CLIP的编码器因低分辨率预训练和依赖噪声较大的网络爬取图文对,在密集空间任务中易丢失细节。为此,我们提出FineViT,一种专为解锁细粒度感知设计的新视觉编码器。通过用密集重描述数据替代粗糙网络数据,采用渐进式训练范式:首先在数十亿全球重描述图文对上从头训练高分辨率编码器,建立丰富的语义基础;随后利用自研的FineCap-450M数据集(含超4.5亿条高质量局部描述)进行大模型对齐,强化局部感知。大量实验验证该策略有效性。FineViT在零样本识别与检索任务中表现卓越,尤其在长上下文检索中领先,集成至MLLM后持续优于SigLIP2与Qwen-ViT等主流多模态视觉编码器。我们希望FineViT能成为细粒度视觉感知的新基准。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) have experienced rapid advancements, their visual encoders frequently remain a performance bottleneck. Conventional CLIP-based encoders struggle with dense spatial tasks due to the loss of visual details caused by low-resolution pretraining and the reliance on noisy, coarse web-crawled image-text pairs. To overcome these limitations, we introduce FineViT, a novel vision encoder specifically designed to unlock fine-grained perception. By replacing coarse web data with dense recaptions, we systematically mitigate information loss through a progressive training paradigm.: first, the encoder is trained from scratch at a high native resolution on billions of global recaptioned image-text pairs, establishing a robust, detail rich semantic foundation. Subsequently, we further enhance its local perception through LLM alignment, utilizing our curated FineCap-450M dataset that comprises over $450$ million high quality local captions. Extensive experiments validate the effectiveness of the progressive strategy. FineViT achieves state-of-the-art zero-shot recognition and retrieval performance, especially in long-context retrieval, and consistently outperforms multimodal visual encoders such as SigLIP2 and Qwen-ViT when integrated into MLLMs. We hope FineViT could serve as a powerful new baseline for fine-grained visual perception.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。