用语言引导视觉推理,让模型更聪明且更轻量。
Language-Guided Abstraction for Visual Reasoning

- 引入语言引导的特权信息分支,融合语义压缩与跨注意力对齐。
- 在ARC任务上达到新最好效果,模型仅1800万参数。
- 适合追求高效通用视觉推理的AI研究者与工程师。
抽象与推理语料库(ARC)被视为通往通用人工智能(AGI)的关键路径,因其使模型能从少量示例中学习抽象变换规则并泛化至新任务。然而,现有方法要么完全依赖大语言模型(纯语言),消耗数十亿参数;要么仅基于视觉(如VARC),难以捕捉高层语义,易陷入像素级模式过拟合。为此,我们提出L-VARC框架,通过语言引导的使用特权信息(LUPI)分支增强视觉推理。具体地,将统一的、任务无关提示输入DeepSeek-V3,构建语义压缩模块,显著优化原始LARC(众包语言描述数据集),适配标准文本编码器(如CLIP)的上下文长度限制。同时设计交叉注意力投影器,对齐视觉特征与语义嵌入,指导模型训练。值得注意的是,该LUPI分支仅用于训练,推断时丢弃,从而获得仅含1800万参数的轻量模型。大量实验表明,L-VARC有效利用语言先验提升视觉推理性能,优于当前最优方法。消融实验证实两项新设计对框架贡献显著。代码已公开于https://github.com/GZHU-DVL/L-VARC。
原文摘要 · Abstract (English)
The Abstraction and Reasoning Corpus (ARC) is viewed as a critical avenue to Artificial General Intelligence (AGI), as it enables models to learn abstract transformation rules from few-shot examples and then generalize to new tasks. However, prevalent ARC methodology is either pure language or vision-only (i.e., VARC). The former depends heavily on LLMs, consuming billions of parameters. The latter often struggles to capture high-level semantics, leading to overfitting on pixel-level patterns. To bridge this gap, we propose L-VARC, a novel framework that enhances visual reasoning via a language-guided Learning Using Privileged Information (LUPI) branch. Specifically, we design a Semantic Compression Module by feeding a unified, task-agnostic prompt into DeepSeek-V3. In this way, the raw LARC (a crowd-sourced language description dataset) can be substantially refined and structured, fitting with the context length constraint of standard text encoders (e.g., CLIP). Moreover, we design a Cross-Attention Projector to align visual features with semantic embeddings, aiming to guide the training of the ARC model. Notably, the LUPI branch is taken in the training process and will be discarded during inference, thereby yielding a lightweight model with a mere 18 million parameters. Extensive experiments demonstrate that our L-VARC effectively leverages linguistic priors to boost visual reasoning and outperforms state-of-the-art. Ablation studies further confirm the contribution of the two new designs towards the L-VARC framework. The code is available at https://github.com/GZHU-DVL/L-VARC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。