提升视觉模型对局部区域的理解能力，增强图像识别与文字检测效果。
Region-based Cluster Discrimination for Visual Representation Learning
- 构建十亿级候选区域数据集，用区域变换器提取细粒度语义。
- 设计统一的聚类判别损失，在单一框架中联合学习物体与文字识别。
- 在分割、密集检测等任务上表现超越此前方法，适合多模态大模型应用。
视觉表征学习是众多下游任务的基础。尽管近年来如CLIP和SigLIP等视觉语言对比模型通过大规模跨模态对齐实现了出色的零样本性能，但其依赖全局表征的特性限制了在密集预测任务（如定位、OCR、分割）中的效果。为此，本文提出区域感知聚类判别方法RICE，首先构建一个十亿级候选区域数据集，并引入区域变换器层以提取丰富的区域语义；进一步设计统一的区域聚类判别损失，可在单个分类框架内联合支持物体与OCR学习，实现大规模数据上的高效可扩展分布式训练。大量实验表明，RICE在分割、密集检测及多模态大语言模型视觉理解任务中持续优于先前方法。预训练模型已开源于https://github.com/deepglint/MVT。
原文摘要 · Abstract (English)
Learning visual representations is foundational for a broad spectrum of downstream tasks. Although recent vision-language contrastive models, such as CLIP and SigLIP, have achieved impressive zero-shot performance via large-scale vision-language alignment, their reliance on global representations constrains their effectiveness for dense prediction tasks, such as grounding, OCR, and segmentation. To address this gap, we introduce Region-Aware Cluster Discrimination (RICE), a novel method that enhances region-level visual and OCR capabilities. We first construct a billion-scale candidate region dataset and propose a Region Transformer layer to extract rich regional semantics. We further design a unified region cluster discrimination loss that jointly supports object and OCR learning within a single classification framework, enabling efficient and scalable distributed training on large-scale data. Extensive experiments show that RICE consistently outperforms previous methods on tasks, including segmentation, dense detection, and visual perception for Multimodal Large Language Models (MLLMs). The pre-trained models have been released at https://github.com/deepglint/MVT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。