通过奖励设计分离多模态模型的感知与推理，发现感知是瓶颈。
Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design
- 设计可验证奖励，让模型在推理中引用图像内容并自我反思。
- 替换图像为文本描述后，Claude模型平均提升20分以上。
- 奖励设计需关注感知误差源头，适合改进视觉推理任务。
基于可验证奖励的强化学习推动了大语言模型推理能力的显著提升，直观上该方法也应适用于多模态模型。然而，多模态模型同时完成图像感知和推理判断，两者联合优化难以区分各自潜力。我们在算法类视觉谜题上研究此问题，发现感知才是主要瓶颈而非推理。将图像替换为简单文本描述后，Claude模型平均性能提升超过20点。进一步评估六种奖励设计，旨在不依赖思维链监督的情况下引导视觉关联。使用GRPO训练Qwen-2.5-VL-7B，诱导出长且结构化的推理过程，包含自我反思与视觉引用，较基础模型提升5.56点。但增益分布不均：无单一奖励覆盖所有类别；带有可验证准确率信号的奖励虽提升域内表现，却牺牲了跨域泛化能力。结果表明，感知意识型奖励设计是未来方向——在感知源头纠正错误，而非在继承错误的推理阶段修正。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards has driven major gains in LLM reasoning, and it is intuitive to assume this recipe will transfer well to multimodal models. However, multimodal models do two things: first, perceive what is in an image, then reason about what it implies. Because these stages are graded jointly, it is hard to tell how much room reasoning alone has to grow. We study this on algorithmic visual puzzles, where both components are necessary and show that perception, not reasoning, is the binding constraint. Replacing images with simple textual descriptions raises performance by over 20 points on average for Claude models. We then evaluate six reward designs aimed at inducing visual grounding during reasoning without chain-of-thought supervision. Training Qwen-2.5-VL-7B with GRPO, reward design induces long, structured reasoning with self-reflection and visual references, yielding a 5.56-point gain over the base model. These gains are, however, uneven; no single reward improves all categories, and rewards with verifiable accuracy signals trade out-of-domain transfer for in-domain accuracy. These results point to perception-aware reward design as a path forward, so that signals correct perception at its source rather than the reasoning that inherits its errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。