arXiv:2510.01304cs.AIcs.CL2025-10被引 11

通过交互式拼图学习,提升视觉语言模型的感知与推理能力。

Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models

  • 将拼图任务设计为模型与环境交互的迭代过程,动态生成动作代码并接收视觉反馈。
  • 在2×2拼图任务中准确率从9.5%提升至82.8%,跨9个视觉任务平均提升3.1%。
  • 无需大量标注数据,提供可扩展的多模态强化学习训练新范式,适合研究通用视觉推理者。

当前大型视觉语言模型(VLMs)虽在多模态理解与推理上取得进展,但其基础感知与推理能力仍受限。例如,在简单拼图任务中,现有模型表现接近随机,暴露出核心能力缺陷。尽管高质量多模态数据可增强性能,但其稀缺且难以规模化。为此,我们提出AGILE——一种用于增强视觉语言模型感知与推理能力的代理式拼图交互学习方法。该方法将拼图求解建模为交互过程:模型每一步生成可执行代码以执行动作,环境则提供细粒度视觉反馈以指导任务完成。通过观察与交互的循环迭代,模型在探索与反馈中逐步提升感知与推理能力。实验表明,AGILE不仅显著提升不同复杂度拼图任务的性能(如2×2设置下准确率从9.5%升至82.8%),还在9个通用视觉任务上实现平均3.1%的性能增益,证明其在感知与推理上的显著增强。本工作为多模态模型的推理与泛化能力发展开辟新路径,并提供了高效、可扩展的多模态强化学习数据解决方案。代码与数据集见:https://github.com/yuzeng0-0/AGILE。

原文摘要 · Abstract (English)

Although current large Vision-Language Models (VLMs) have advanced in multimodal understanding and reasoning, their fundamental perceptual and reasoning abilities remain limited. Specifically, even on simple jigsaw tasks, existing VLMs perform near randomly, revealing deficiencies in core perception and reasoning capabilities. While high-quality vision-language data can enhance these capabilities, its scarcity and limited scalability impose significant constraints. To address this, we propose AGILE, an Agentic jiGsaw Interaction Learning for Enhancing visual perception and reasoning in VLMs. AGILE formulates jigsaw solving as an interactive process, enabling the model to progressively engage with the environment. At each step, the model generates executable code to perform an action based on the current state, while the environment provides fine-grained visual feedback to guide task completion. Through this iterative cycle of observation and interaction, the model incrementally improves its perceptual and reasoning capabilities via exploration and feedback. Experimental results show that AGILE not only substantially boosts performance on jigsaw tasks of varying complexity (e.g., increasing accuracy from 9.5% to 82.8% under the 2 $\times$ 2 setting) but also demonstrates strong generalization across 9 general vision tasks, achieving an average improvement of 3.1%. These results indicate notable enhancements in both perceptual and reasoning abilities. This work opens a new avenue for advancing reasoning and generalization in multimodal models and provides an efficient, scalable solution to the scarcity of multimodal reinforcement learning data. The code and datasets is available at https://github.com/yuzeng0-0/AGILE .

视觉推理多模态学习强化学习生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。