通过分阶段数据与采样策略,提升模型在知识型视觉问答中的推理能力。
Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum
- 设计可调控难度的数据生成和课程采样策略,逐步引导模型学习。
- 在Encyclopedic VQA和InfoSeek上分别提升准确率至37.1%和44.1%。
- 适合研究多模态推理、知识融合与强化学习的开发者参考。
知识型视觉问答(KB-VQA)要求模型结合外部知识回答图像相关问题,因检索噪声和知识库结构化、百科式特性,导致其分布与预训练多模态大模型存在差距,使后训练阶段的推理与领域适配困难。本文提出基于数据生成的课程强化学习框架Wiki-R1,系统性激励多模态大模型在KB-VQA任务中进行推理。Wiki-R1构建一系列与模型能力演进对齐的训练分布,弥合从预训练到目标分布的差距。提出可控课程数据生成方法,调节检索器生成不同难度样本;设计课程采样策略,选取在强化学习更新中可能带来非零收益的有信息量样本。通过观测奖励估计样本难度,并传播至未观测样本以指导学习。在两个KB-VQA基准测试Encyclopedic VQA和InfoSeek上实验表明,Wiki-R1达到新最佳性能,准确率分别从35.5%提升至37.1%,从40.1%提升至44.1%。
原文摘要 · Abstract (English)
Knowledge-Based Visual Question Answering (KB-VQA) requires models to answer questions about an image by integrating external knowledge, posing significant challenges due to noisy retrieval and the structured, encyclopedic nature of the knowledge base. These characteristics create a distributional gap from pretrained multimodal large language models (MLLMs), making effective reasoning and domain adaptation difficult in the post-training stage. In this work, we propose \textit{Wiki-R1}, a data-generation-based curriculum reinforcement learning framework that systematically incentivizes reasoning in MLLMs for KB-VQA. Wiki-R1 constructs a sequence of training distributions aligned with the model's evolving capability, bridging the gap from pretraining to the KB-VQA target distribution. We introduce \textit{controllable curriculum data generation}, which manipulates the retriever to produce samples at desired difficulty levels, and a \textit{curriculum sampling strategy} that selects informative samples likely to yield non-zero advantages during RL updates. Sample difficulty is estimated using observed rewards and propagated to unobserved samples to guide learning. Experiments on two KB-VQA benchmarks, Encyclopedic VQA and InfoSeek, demonstrate that Wiki-R1 achieves new state-of-the-art results, improving accuracy from 35.5\% to 37.1\% on Encyclopedic VQA and from 40.1\% to 44.1\% on InfoSeek. The project page is available at https://artanic30.github.io/project_pages/WikiR1/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。