让多模态大模型通过自我想象迭代修改视觉输入,突破感知瓶颈。
Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Models
- 提出自主想象机制,让模型自动生成中间视觉状态
- 无需重训练即可解决原能力外的计数与拼图任务
- 适合研究视觉推理与闭环生成的学者参考
在纯文本模态下,大语言模型通过分解复杂任务为子问题,展现出强大推理能力。然而,多模态大语言模型(MLLMs)在看似简单的视觉任务(如计数、拼图)上仍表现不佳。我们认为,这源于视觉到文本转换的局限性:当输入视觉信息超出模型感知能力时,仅靠扩大推理规模无法突破该瓶颈。为此,我们提出「自主想象」方法,使MLLMs可迭代修改视觉输入(如隔离物体、重排拼图块),形成闭合环路的视觉修改步骤,将视觉转文本过程分解为可解的子步骤。实验表明,无需任何微调,模型即可解决原本超出其感知能力的任务,证明闭环视觉修改是有效分解视觉推理的新路径。代码与数据已开源。
原文摘要 · Abstract (English)
Under pure textual modality, Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning tasks by decomposing them into simpler sub-problems. However, Multimodal Large Language Models (MLLMs) still struggle with some seemingly straightforward visual tasks, such as counting and solving jigsaw puzzles. We argue that these tasks challenge the ability of visual-to-textual conversion, where MLLMs convert visual information perceived from the input scene, to textual information for further reasoning and generating the answer. If the complexity of the visual input is beyond the perceptual capability of the MLLMs, without decomposing this conversion process, simply scaling inference-time reasoning cannot solve the task because it repeatedly encounters the same perceptual bottleneck. We propose an approach, autonomous imagination, to enable MLLMs to iteratively modify visual inputs (e.g. isolating objects, rearranging puzzle pieces) into intermediate visual states, decomposing visual-to-textual conversion into closed-loop visual modification steps. We show that, without any retraining, MLLMs can now solve tasks initially beyond their perceptual capability, highlighting that closed-loop visual modification can be an effective way of decomposing the visual reasoning task into solvable substeps. Our code and data are released at https://future-item.github.io/autoimagine-site/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。