arXiv:2606.26794cs.CVcs.AI2026-06中稿 · ECCV

给CLIP模型注入常识推理能力,提升视觉理解深度。

ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

论文配图:ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP
图 1 · 摘自论文原文
  • 两阶段持续预训练,逐步加入推理信号并保留图像描述对齐
  • 在5800万样本上训练,零样本检索性能提升12.3%(绝对值)
  • 适合需要强视觉推理的多模态大模型,如LLaVA-NeXT

CLIP及其变体广泛用作多模态系统的视觉主干,但其预训练仍以图像-文本描述对齐为主。随着下游应用对视觉化常识推理和组合推理需求增加,现有方法是否能在不改变架构的前提下支持此类推理仍不明确。为此,我们提出ReasonCLIP-58M,一种持续预训练框架,通过两阶段策略将大规模推理监督引入CLIP式模型:先渐进式融合推理信号,同时保持描述性对齐;再进行类别结构化的推理监督。为支持该框架,我们构建了两个互补数据集与基准:ReasonLite-42M(开放形式、可视觉验证的推理描述),ReasonPro-16M(类别特定推理监督)及RCLIP-Bench用于诊断评估视觉化推理能力。我们训练出一系列ReasonCLIP模型,在增强视觉化常识与组合推理能力的同时,显著提升零样本检索性能。作为多模态大语言模型(如LLaVA-NeXT)的即插即用视觉编码器,ReasonCLIP在无额外推理成本下带来一致增益,表明结构化推理监督能有效提升CLIP式视觉表征的表达能力。所有数据集、模型与训练代码已开源于https://github.com/RISys-Lab/ReasonCLIP。

原文摘要 · Abstract (English)

CLIP and its variants are widely adopted visual backbones in multimodal systems, but their pretraining remains dominated by descriptive image-text alignment. As downstream applications increasingly demand visually grounded commonsense inference and compositional reasoning, it remains unclear whether CLIP-style encoders can support such reasoning without architectural changes. To address this, we present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models through our two-stage strategy, which progressively integrates reasoning signals while preserving descriptive alignment, followed by category-structured reasoning supervision. To support this framework, we construct two complementary datasets and a benchmark: ReasonLite-42M, with open-form, visually verifiable reasoning captions; ReasonPro-16M, with category-specific reasoning supervision; and RCLIP-Bench for diagnostic evaluation of visually grounded reasoning. We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance. As a drop-in visual encoder for multimodal large language models such as LLaVA-NeXT, ReasonCLIP delivers consistent gains without additional inference cost, demonstrating that structured reasoning supervision enhances the expressive capacity of CLIP-style visual representations. All datasets, models, and training code are available at https://github.com/RISys-Lab/ReasonCLIP.

视觉推理CLIP多模态常识推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。