让视频理解模型学会判断每帧可靠性,提升真实场景下的推理鲁棒性。
Confidence-Aware Tool Orchestration for Robust Video Understanding

- 引入可信度评分筛选可靠帧,驱动多工具协同推理
- 在8个任务上清输入下达56.4%准确率,比最强开源基线高10.6个百分点
- 面对五类真实干扰仍保持54.3%准确率,误差下降最小,适合部署于复杂环境
视频推理大模型隐含假设所有输入帧可靠性相同,导致在运动模糊、反光或遮挡等现实扰动下,前沿模型在真实世界具身基准上准确率下降15-30个百分点,却无法察觉视觉证据质量退化。为此,我们提出Robust-TO框架,将每帧可信度显式融入推理全流程。该框架统一管理异构视觉感知工具,每个工具接收由原问题分解出的子查询及由可靠性-相关性得分筛选的可信帧,返回统一格式的证据:具体预测(如边界框、运动轨迹、识别文本或动作标签)、时间定位与校准后的可靠性分数。推理过程中,这些分数指导三层级证据融合(高/中/低),并定义置信度-成本GRPO奖励,联合优化正确性、证据可靠性与效率。在涵盖八个任务的两个视频推理基准上,Robust-TO在干净输入下达到56.4%平均准确率,优于最强开源基线10.6个百分点,超过Gemini-2.5-Pro(46.2%)。在五类真实畸变下,仍保持54.3%平均准确率,高于最强开源基线5.8个百分点,且在所有对比方法中表现出最小的清洁到扰动准确率下降。
原文摘要 · Abstract (English)
Video reasoning language models implicitly assume that every input frame is equally reliable. This leads to what we term the Blind Trust Problem: under realistic perturbations such as motion blur, glare, or occlusion, frontier video reasoning models can suffer 15-30%p accuracy drops on real-world embodied benchmarks, while remaining unaware that their visual evidence has been degraded. To address this challenge, we propose Robust-TO, an agentic video understanding framework that explicitly integrates per-frame trustworthiness into every stage of reasoning. Robust-TO organizes heterogeneous visual perception tools under a unified evidence interface. Each tool receives a sub-query derived from the original question and a set of trustworthy frames selected by the reliability-relevance score. It returns evidence in a shared format: a concrete prediction (e.g., a bounding box, motion trajectory, recognized text, or action label), temporal grounding, and a calibrated reliability score. During reasoning, these calibrated scores guide evidence weighting in a three-tier synthesis process (high/medium/low) and define a confidence-cost GRPO reward that jointly optimizes correctness, evidence reliability, and efficiency. On two video reasoning benchmarks spanning eight tasks, Robust-TO achieves 56.4% average accuracy on clean inputs, surpassing the strongest open-source baseline by 10.6%p and outperforming Gemini-2.5-Pro (46.2%). Under five realistic corruption types, Robust-TO maintains 54.3% average accuracy, 5.8%p above the strongest open-source baseline, while exhibiting the smallest clean-to-corrupted accuracy drop among all compared methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。