用统一视觉框架处理复杂网页图文,无需文本解析或配对数据。
Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documents

- 将文本、图像全转为像素输入,用单个视觉模型统一建模。
- 在三个新基准上表现超越或媲美CLIP,尤其在细粒度顺序理解上优势明显。
- 适合处理真实网页中交错图文的场景,开源代码已发布。
对比学习型多模态模型如CLIP在大量任务中表现出色,但面对真实网页中交错、松散对齐或视觉嵌入的图文时仍受限。为此,我们提出视觉中心对比学习(VC2L),一种统一框架,使用单一视觉变压器建模文本、图像及其组合。VC2L完全在像素空间运行,将所有输入(文本、图像或混合)渲染为图像,避免了OCR、文本分词或模态融合策略。为捕捉网页文档中的复杂跨模态关系,VC2L采用片段级对比学习目标,对连续多模态段落进行对齐,利用文档内在连贯性,无需显式图像-文本配对数据。为评估该方法,我们引入三个检索基准:AnyCIR(跨模态检索)、SeqCIR(细粒度顺序理解)和CSR(未见数据泛化)。实验结果表明,VC2L在自建基准及M-BEIR、MTEB等现有数据集上均达到竞争力或更优性能。这些发现凸显了多模态网页数据作为对比学习训练资源的潜力,并展示了统一视觉中心方法在多模态表征学习中的可扩展性。代码与模型已在https://github.com/showlab/VC2L发布。
原文摘要 · Abstract (English)
Contrastive vision-language models such as CLIP have demonstrated strong performance across a wide range of multimodal tasks by learning from aligned image-text pairs. However, their ability to handle complex, real-world web documents remains limited, particularly in scenarios where text and images are interleaved, loosely aligned, or embedded in visual form. To address these challenges, we propose Vision-Centric Contrastive Learning (VC2L), a unified framework that models text, images, and their combinations using a single vision transformer. VC2L operates entirely in pixel space by rendering all inputs, whether textual, visual, or combined, as images, thus eliminating the need for OCR, text tokenization, or modality fusion strategy. To capture complex cross-modal relationships in multimodal web documents, VC2L employs a snippet-level contrastive learning objective that aligns consecutive multimodal segments, leveraging the inherent coherence of documents without requiring explicitly paired image-text data. To assess the effectiveness of this approach, we introduce three retrieval benchmarks, AnyCIR, SeqCIR, and CSR, designed to evaluate cross-modal retrieval, fine-grained sequential understanding, and generalization to unseen data, respectively. Empirical results show that VC2L achieves competitive or superior performance compared to CLIP-style models on both the proposed benchmarks and established datasets such as M-BEIR and MTEB. These findings underscore the potential of multimodal web data as a valuable training resource for contrastive learning and illustrate the scalability of a unified, vision-centric approach for multimodal representation learning. Code and models are available at: https://github.com/showlab/VC2L.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。