用拼图游戏训练大模型看图能力,无需标注也能提升视觉理解。
Visual Jigsaw Post-Training Improves MLLMs
- 把图像视频3D数据打乱后让模型用语言还原顺序
- 在多个任务上显著提升细粒度感知与时空推理能力
- 适合关注视觉理解增强的多模态模型研究者
基于强化学习的后训练已成为提升多模态大模型对齐与推理能力的有效范式。尽管视觉后训练对增强模型对视觉信号的理解至关重要,但现有方法大多以文本为中心,仅将密集视觉输入作为稀疏提示用于文本推理。少数视觉导向方法仍依赖文本中介或引入额外生成结构。本文提出 Visual Jigsaw,一种通用自监督后训练框架,旨在强化多模态大模型的视觉理解。该框架将视觉输入分割、打乱,要求模型通过自然语言生成正确排列顺序,形成通用排序任务。此设计天然适配可验证奖励的强化学习(RLVR),无需额外生成组件,且自动获得监督信号而无需人工标注。我们在图像、视频和3D数据三种模态上验证了该方法。大量实验表明,模型在细粒度感知、时间推理和3D空间理解方面均有显著提升。结果凸显了自监督视觉中心任务在后训练中的潜力,鼓励未来对视觉中心预训练任务的设计探索。项目页面:https://penghao-wu.github.io/visual_jigsaw/
原文摘要 · Abstract (English)
Reinforcement learning based post-training has recently emerged as a powerful paradigm for enhancing the alignment and reasoning capabilities of multimodal large language models (MLLMs). While vision-centric post-training is crucial for enhancing MLLMs' intrinsic understanding of visual signals, current post-training paradigms are predominantly text-centric, where dense visual inputs are only leveraged to extract sparse cues for text-based reasoning. There exist a few approaches in this direction, however, they often still rely on text as an intermediate mediator or introduce additional visual generative designs. In this work, we introduce Visual Jigsaw, a generic self-supervised post-training framework designed to strengthen visual understanding in MLLMs. Visual Jigsaw is formulated as a general ordering task: visual inputs are partitioned, shuffled, and the model must reconstruct the visual information by producing the correct permutation in natural language. This naturally aligns with reinforcement learning from verifiable rewards (RLVR), requires no additional visual generative components, and derives its supervisory signal automatically without any annotations. We instantiate Visual Jigsaw across three visual modalities, including images, videos, and 3D data. Extensive experiments demonstrate substantial improvements in fine-grained perception, temporal reasoning, and 3D spatial understanding. Our findings highlight the potential of self-supervised vision-centric tasks in post-training MLLMs and aim to inspire further research on vision-centric pretext designs. Project Page: https://penghao-wu.github.io/visual_jigsaw/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。