让模型学会生成可验证证据,推理时通过投票选出最可信答案。
Reinforce to Learn, Elect to Reason: A Dual Paradigm for Video Reasoning
- 训练时用强化学习引导模型生成结构化证据
- 推理时用多候选投票机制提升答案可靠性,平均用3.1个候选
- 适合需要高可信度视频理解的场景,如医疗或自动驾驶
视频推理虽因大模型进步显著,但其推理过程常为单次输出,缺乏对推理与证据一致性的验证。本文提出双范式框架RLER,将生成证据与获取可靠答案解耦。在训练阶段,采用组相对强化学习和三项任务驱动奖励:帧敏感奖励使推理锚定关键帧,思维透明性奖励生成可读可解析的推理链,反重复奖励提升信息密度。这些信号促使模型输出机器可验证的结构化证据。在推理阶段,采用无需训练的调度器生成少量多样化候选,解析其答案与引用帧,基于证据一致性、置信度、透明性和非冗余性评分,并进行鲁棒的加权选举。该闭环设计提升了可靠性与可解释性,且不增加模型规模。在8个基准上全面评估,相较基线模型平均提升6.3%,每题平均使用3.1个候选,实现计算开销与质量的良好平衡。结果支持一个核心观点:训练中显式生成证据,推理中基于证据投票,是构建可信视频推理的有效路径。
原文摘要 · Abstract (English)
Video reasoning has advanced with large multimodal models (LMMs), yet their inference is often a single pass that returns an answer without verifying whether the reasoning is evidence-aligned. We introduce Reinforce to Learn, Elect to Reason (RLER), a dual paradigm that decouples learning to produce evidence from obtaining a reliable answer. In RLER-Training, we optimize the policy with group-relative reinforcement learning (RL) and 3 novel task-driven rewards: Frame-sensitive reward grounds reasoning on explicit key frames, Think-transparency reward shapes readable and parsable reasoning traces, and Anti-repetition reward boosts information density. These signals teach the model to emit structured, machine-checkable evidence and potentiate reasoning capabilities. In RLER-Inference, we apply a train-free orchestrator that generates a small set of diverse candidates, parses their answers and cited frames, scores them by evidence consistency, confidence, transparency, and non-redundancy, and then performs a robust evidence-weighted election. This closes the loop between producing and using evidence, improving reliability and interpretability without enlarging the model. We comprehensively evaluate RLER against various open-source and RL-based LMMs on 8 representative benchmarks. RLER achieves state of the art across all benchmarks and delivers an average improvement of 6.3\% over base models, while using on average 3.1 candidates per question, indicating a favorable balance between compute and quality. The results support a simple thesis: making evidence explicit during learning and electing by evidence during inference is a robust path to trustworthy video reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。