让大模型自己发现视觉推理线索,无需人工标注中间步骤。
Latent Implicit Visual Reasoning
- 通过隐式潜变量自动挖掘视觉推理线索,不依赖人工标注。
- 在多个视觉任务上超越直接微调,且适配不同大模型架构。
- 无需辅助图像、边界框或思维链标注,通用性强适合多场景。
尽管大型多模态模型(LMMs)取得了显著进展,但它们仍以语言为中心,主要依赖语言进行推理,难以处理以视觉为主的任务。现有方法通过辅助图像、深度图或图像裁片监督中间视觉步骤来弥补这一缺陷,但这些方法施加了严格的先验假设,增加了标注成本,且泛化能力差。为此,我们提出隐式潜在视觉推理(LIVR),一种任务无关的机制,训练LMMs自主发现并使用潜变量视觉推理标记,无需显式中间监督。这些标记全局关注并以任务自适应方式重新编码图像,使模型能提取相关视觉信息而无需人工设计的监督。LIVR在多种以视觉为核心的任务中持续优于直接监督微调,并在与已有文本基及显式视觉中间推理方法的对比中保持竞争力或更优表现,同时无需额外的中间监督(如辅助图像、边界框、图像裁片、深度图或思维链标注)。项目主页见:https://www.chuyishang.com/livr/
原文摘要 · Abstract (English)
While Large Multimodal Models (LMMs) have made significant progress, they remain largely text-centric, relying on language as their core reasoning modality. As a result, they are limited in their ability to handle reasoning tasks that are predominantly visual. Recent approaches have sought to address this by supervising intermediate visual steps with helper images, depth maps, or image crops. However, these strategies impose restrictive priors on what "useful" visual abstractions look like, add heavy annotation costs, and struggle to generalize across tasks. To address this critical limitation, we propose Latent Implicit Visual Reasoning (LIVR), a task-agnostic mechanism that trains LMMs to discover and use latent visual reasoning tokens without explicit intermediate supervision. These tokens attend globally and re-encode the image in a task-adaptive way, enabling the model to extract relevant visual information without hand-crafted supervision. LIVR consistently outperforms direct supervised fine-tuning across diverse vision-centric tasks and multiple LMM backbones. In broader comparisons, LIVR remains competitive with or outperforms prior text-based and explicit-visual-intermediate reasoning methods, while requiring no additional intermediate supervision such as helper images, bounding boxes, image crops, depth maps, or chain-of-thought annotations. Our project page can be found here: https://www.chuyishang.com/livr/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。