arXiv:2606.27373cs.CV2026-06被引 3

让AI模型更关注图像内容,提升视觉理解能力。

Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

论文配图:Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
图 1 · 摘自论文原文
  • 设计双激励机制,强制模型在生成时聚焦图像特征。
  • 在COCO和TextCaps上分别提升16.85和19.66的CIDEr分数。
  • 无需标注、角色或外部模型,适用于多种大模型架构。

近期自演化大型多模态模型(LMMs)在纯无监督设置下提升了视觉推理能力。然而,现有方法中的多角色自对弈与自一致性奖励机制仅优化答案一致性,未确保解码器关注视觉内容,导致模型依赖语言统计先验而非图像生成,产生视觉条件不足问题,表现为对视觉标记关注度低。这使得当前自演化LMMs在图像描述和视觉问答等任务中表现不佳。为此,我们提出VISE(视觉不变性自演化),一种纯无监督框架,通过两种互补的不变性奖励直接正则化模型的视觉条件策略:几何不变性奖励要求在已知变换下保持空间一致性;语义不变性奖励通过要求模型在扰动预测区域时识别证据缺失,惩罚无依据生成。VISE在单个模型内运行,无需专用角色、外部奖励模型或标注,训练数据为原始未标注图像。在18个基准测试中验证有效:以Qwen3-VL-2B为基础,VISE在COCO上提升+16.85 CIDEr,TextCaps上提升+19.66 CIDEr,对象幻觉减少5.0 Chair-I点,并跨四个模型家族与规模泛化。代码与模型已公开。

原文摘要 · Abstract (English)

Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision--language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18 benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE

多模态自演化视觉推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。