让视觉语言模型自己提问,分步推理提升复杂视觉问答准确率。
Self-Questioning Vision-Language Models: Reinforcement Learning for Compositional Visual Reasoning

- 模型通过强化学习自动拆分问题为子问题,逐步求解。
- 在A-OKVQA上准确率提升至52.2%,优于基线46.8%。
- 适合需要多步推理的复杂视觉理解任务。
视觉语言模型(VLMs)虽能处理图像与文本,但在需多步推理的组合式视觉问题上表现不佳,如识别物体、计数并比较。现有方法依赖人工编写的分步解释训练,成本高且难扩展。本文提出自提问框架,让一个30亿参数的VLM通过强化学习(GRPO)自动将视觉问题分解为子问题,逐个回答后得出最终答案。模型从未见过分解示例,仅根据奖励信号——包含子问题数量和答案正确性——自我优化。在合成几何场景(CLEVR)和真实照片(A-OKVQA)上训练。在A-OKVQA上,自提问和标准强化学习分别将准确率提升至52.2%和51.6%,优于未训练模型的46.8%。首次通过奖励中间子问题生成,使模型自发发现组合式分解策略。结果表明,引导AI自我提问是应对复杂视觉推理的有效路径,尤其适用于需显式分步解析的问题。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are AI systems that process both images and text, yet they often struggle with compositional visual reasoning questions that require chaining multiple steps together, such as identifying objects, counting them, and comparing the results. Existing approaches improve this reasoning by training models on human-written step-by-step explanations, but creating these annotations is expensive and difficult to scale. We propose a self-questioning framework that trains a VLM to break visual questions into smaller sub-questions and answer each one before producing a final response, using a reinforcement learning algorithm called Group Relative Policy Optimization (GRPO). The model is never shown examples of how to decompose questions, it discovers this behavior on its own, guided by a reward signal that scores whether the output contains sub-questions and whether the final answer is correct. We apply this framework to a 3-billion-parameter model, training on both synthetic scenes of geometric shapes (CLEVR) and real-world photographs (A-OKVQA). On A-OKVQA, both self-questioning and standard reinforcement learning substantially improve accuracy over the untrained model (52.2% and 51.6% vs. 46.8%). We introduce the first self-questioning VLM by rewarding not only the final answer like standard RL but additionally for generating intermediate sub-questions, enabling it to discover compositional decomposition strategies. These results suggest that teaching AI systems to ask themselves intermediate questions is a promising strategy for complex visual reasoning, particularly when the difficulty of a question warrants explicit step-by-step decomposition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。