arXiv:2605.05922cs.CV2026-05

提出解耦推理与打分的视频奖励模型,提升泛化能力与训练稳定性。

Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling

论文配图:Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling
图 1 · 摘自论文原文
  • 先生成思维链,再由专用模块打分,分离推理与评分过程。
  • 两阶段训练:冷启动确保打分鲁棒性,强化学习独立优化推理与评分。
  • 相比联合推理打分,更少依赖数据量,适合复杂视频评估场景。

生成式视频模型的进步越来越依赖于训练后和测试时的扩展,而这都严重依赖视频奖励模型(RMs)的质量。理想的奖励模型应能准确预测人类偏好下的奖励,且在各种场景中保持一致。然而现有方法面临根本困境:判别式奖励模型直接对多模态大语言模型(MLLM)提取的特征进行奖励回归,缺乏显式推理,易导致捷径学习,并严重依赖大规模数据以实现泛化;而带有思维链(CoT)推理的生成式奖励模型虽具备更好的可解释性和泛化潜力,但因推理与评分耦合在单一自回归推理链中,存在固有的优化瓶颈。为兼顾CoT推理的泛化优势并缓解联合推理与评分的训练不稳定性,我们提出DeScore,一种高效训练且具有良好泛化性的视频奖励模型。DeScore采用解耦的‘思考-打分’范式:首先由MLLM生成显式思维链,随后由一个专用的判别式打分模块(包含可学习查询令牌和回归头)预测最终奖励。DeScore通过两阶段框架优化:(1) 判别式冷启动阶段引入随机掩码机制,确保打分能力稳健;(2) 双目标强化学习阶段,独立优化思维链质量并校准最终奖励,确保更高质量的推理直接带来更高性能。

原文摘要 · Abstract (English)

Recent advances in generative video models are increasingly driven by post-training and test-time scaling, both of which critically depend on the quality of video reward models (RMs). An ideal reward model should predict accurate rewards that align with human preferences across diverse scenarios. However, existing paradigms face a fundamental dilemma: \textit{Discriminative RMs} regress rewards directly on features extracted by multimodal large language models (MLLMs) without explicit reasoning, making them prone to shortcut learning and heavily reliant on massive data scaling for generalization. In contrast, \textit{Generative RMs} with Chain-of-Thought (CoT) reasoning exhibit superior interpretability and generalization potential, as they leverage fine-grained semantic supervision to internalize the rationales behind human preferences. However, they suffer from inherent optimization bottlenecks due to the coupling of reasoning and scoring within a single autoregressive inference chain. To harness the generalization benefits of CoT reasoning while mitigating the training instability of coupled reasoning and scoring, we introduce DeScore, a training-efficient and generalizable video reward model. DeScore employs a decoupled ``think-then-score'' paradigm: an MLLM first generates an explicit CoT, followed by a dedicated discriminative scoring module consisting of a learnable query token and a regression head that predicts the final reward. DeScore is optimized via a two-stage framework: (1) a discriminative cold start incorporating a random mask mechanism to ensure robust scoring capabilities, and (2) a dual-objective reinforcement learning stage that independently refines CoT reasoning quality and calibrates the final reward, ensuring that higher-quality reasoning directly translates to superior model performance.

视频奖励思维链解耦设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。