arXiv:2505.24718cs.CV2025-05被引 24

让视频推理更聚焦:通过注意力加权和细粒度奖励提升模型表现

Reinforcing Video Reasoning with Focused Thinking

  • 用信息熵筛选关键词元,抑制冗余推理过程
  • 在CLEVRER上达50.4%准确率,较Video-R1提升18.8%
  • 适合需要精准视频理解与复杂推理的研究者

近期基于组相对策略优化(GRPO)的强化学习进展显著提升了多模态大语言模型在复杂推理任务中的表现。然而仍存在两大瓶颈:一是推理链冗长模糊,掩盖关键时空线索;二是二值奖励无法区分部分正确答案,导致奖励方差大、学习效率低。本文提出TW-GRPO框架,通过引入词元加权机制,优先保留高信息密度的词元(基于组内信息熵估计),抑制通用推理前缀等冗余内容;同时将训练从单选转为多选问答,采用软奖励实现更精细的梯度估计,以区分部分正确结果;此外提出问题-答案反转的数据增强策略,从现有基准生成多样化多选样本。实验表明,该方法在多个视频推理与通用理解基准上达到领先性能:在CLEVRER上取得50.4%准确率(相较Video-R1提升18.8%),在MMVU上达65.8%。代码已开源。

原文摘要 · Abstract (English)

Recent advancements in reinforcement learning, particularly through Group Relative Policy Optimization (GRPO), have significantly improved multimodal large language models for complex reasoning tasks. However, two critical limitations persist: 1) they often produce unfocused, verbose reasoning chains that obscure salient spatiotemporal cues and 2) binary rewarding fails to account for partially correct answers, resulting in high reward variance and inefficient learning. In this paper, we propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity. Specifically, we employs a token weighting mechanism that prioritizes tokens with high informational density (estimated by intra-group information entropy), suppressing redundant tokens like generic reasoning prefixes. Furthermore, we reformulate RL training by shifting from single-choice to multi-choice QA tasks, where soft rewards enable finer-grained gradient estimation by distinguishing partial correctness. Additionally, we propose question-answer inversion, a data augmentation strategy to generate diverse multi-choice samples from existing benchmarks. Experiments demonstrate state-of-the-art performance on several video reasoning and general understanding benchmarks. Notably, TW-GRPO achieves 50.4\% accuracy on CLEVRER (18.8\% improvement over Video-R1) and 65.8\% on MMVU. Our codes are available at \href{https://github.com/longmalongma/TW-GRPO}.

视频推理强化学习多模态注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。