为多模态大模型设计分枝策略优化，提升视觉推理准确性
Multi-Branch Policy Optimization for Multimodal Large Language Models
- 构建视觉-语言决策点的推理树，支持多分支并行探索
- 采用分支相对优势进行段级奖励分配，避免信号衰减
- 引入时间回放缓冲区，复用有效推理片段提高训练效率
多模态大模型的分组强化学习通常依赖轨迹级信用分配，对响应中所有词元统一赋予单一优势值。然而，多模态推理面临远高于纯文本场景的感知不确定性，模型需反复审视视觉信息以验证中间推理，不同视觉依据可能导致分歧的推理路径，使得均匀信用分配尤为不适用，导致相对优势逐步退化至零。为此，我们提出多分支策略优化（MBPO），一种基于树结构的框架，在视觉-语言决策边界构建推理树，使兄弟分支可并行探索多样化的视觉假设，并通过分支相对优势实现段级信用分配。我们进一步引入时间回放缓冲区，重用有信息量的推理片段，同时控制策略过时。在多个多模态推理基准上的实验表明，MBPO优于代表性基线，显著提升学习信号质量和优化效率。代码已公开于 https://github.com/ShuaiLyu0110/MBPO。
原文摘要 · Abstract (English)
Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. However, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at https://github.com/ShuaiLyu0110/MBPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。