arXiv:2506.09736cs.CVcs.AI2025-06被引 7

用图像扰动提升视觉理解,让多模态模型更会‘看’。

Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation

  • 通过三种图像扰动增强视觉感知鲁棒性,无需修改算法或额外数据。
  • 在多个数据集上显著提升数学推理性能,效果媲美算法改进。
  • 适合关注多模态模型视觉能力优化的研究者与开发者。

尽管多模态大语言模型(MLLMs)发展迅速,但其对视觉处理的重视不足。一个简单却关键的实验发现,仅依赖语言的模型在获得图像描述后,表现可媲美甚至优于使用原始视觉输入的MLLMs。这表明当前MLLMs虽能生成准确的视觉描述,却未能有效整合用于推理。为此,我们提出一种简单的图像扰动框架,可在不修改算法或增加训练数据的前提下提升感知鲁棒性。该方法包含三种针对性扰动:干扰项拼接、保持主导性的混合、随机旋转,可轻松集成至SFT、DPO和GRPO等后训练流程中。在多个数据集上的大量实验显示,该方法在数学推理任务中实现持续性能提升,增益相当于算法级改进。此外,仅通过图像扰动训练的Qwen2.5-VL-7B在开源7B规模强化学习微调模型中表现优异。全面消融实验揭示不同扰动策略对视觉推理各维度的独特贡献。研究强调了图像扰动在多模态数学推理中的核心作用:更好的推理始于更好的‘看见’。代码已公开于https://github.com/YutingLi0606/Vision-Matters。

原文摘要 · Abstract (English)

Despite the rapid progress of multimodal large language models (MLLMs), they have largely overlooked the importance of visual processing. In a simple yet revealing experiment, we interestingly find that language-only models, when provided with image captions, can achieve comparable or even better performance than MLLMs that consume raw visual inputs. This suggests that current MLLMs may generate accurate visual descriptions but fail to effectively integrate them during reasoning. Motivated by this, we propose a simple visual perturbation framework that enhances perceptual robustness without requiring algorithmic modifications or additional training data. Our approach introduces three targeted perturbations: distractor concatenation, dominance-preserving mixup, and random rotation, that can be easily integrated into existing post-training pipelines including SFT, DPO, and GRPO. Through extensive experiments across multiple datasets, we demonstrate consistent improvements in mathematical reasoning performance, with gains comparable to those achieved through algorithmic changes. Additionally, we achieve competitive performance among open-source 7B RL-tuned models by training Qwen2.5-VL-7B with visual perturbation. Through comprehensive ablation studies, we analyze the effectiveness of different perturbation strategies, revealing that each perturbation type contributes uniquely to different aspects of visual reasoning. Our findings highlight the critical role of visual perturbation in multimodal mathematical reasoning: better reasoning begins with better seeing. Our code is available at https://github.com/YutingLi0606/Vision-Matters.

多模态视觉推理扰动

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。