让视觉推理模型学会动态切换看图和想文,提升思考准确性
DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning

- 用注意力变化判断每个词该偏视觉还是文本,动态分配角色
- 在7个基准上,4种强化学习方法均显著提升推理准确率
- 适用于各类视觉语言模型,无需修改原有算法
强化学习结合可验证奖励(RLVR)已成为提升多模态大语言模型(MLLMs)视觉推理能力的主流方法。然而,现有方法主要优化最终推理结果,忽视生成过程中的细粒度跨模态协调。通过逐词分析与受控干预,我们发现,在思维链(CoT)推理中,MLLMs常无法动态交替提取视觉证据与生成文本上下文,这种协调失效与推理失败直接相关。为此,我们提出DyCo-RL,将动态跨模态协调机制融入RLVR优化。具体地,DyCo-RL利用Fisher-Rao测地距离衡量模态内注意力转移,为每个词分配视觉导向或文本导向的功能角色,并评估其实际注意力分配与角色的对齐程度,以此指导策略优化中的优势重加权。大量实验表明,该算法无关的DyCo-RL应用于Qwen2.5-VL-3B/7B时,可在七个涵盖视觉中心与数学推理的基准上,持续提升四种代表性RLVR算法的性能。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a leading paradigm for enhancing visual reasoning in Multimodal Large Language Models (MLLMs). However, existing RLVR methods optimize primarily for the reasoning outcome, fundamentally overlooking the fine-grained cross-modal coordination required during the generation process. Through token-level analyses and controlled interventions, we reveal that during Chain-of-Thought (CoT) reasoning, MLLMs frequently fail to dynamically alternate between extracting visual evidence and synthesizing textual context-a coordination breakdown that is causally linked to reasoning failures. Motivated by these findings, we propose DyCo-RL, which integrates dynamic cross-modal coordination into RLVR optimization. Specifically, DyCo-RL uses the Fisher-Rao geodesic distance to measure within-modality attention shifts, assigning tokens to either visually-oriented or text-oriented functional roles. It then evaluates the alignment between a token's actual attention allocation and its assigned role, leveraging this score for alignment-guided advantage reweighting during policy optimization. Extensive experiments demonstrate that the algorithm-agnostic DyCo-RL, when applied to Qwen2.5-VL-3B/7B, consistently improves four representative RLVR algorithms across seven benchmarks spanning visual-centric and mathematical reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。