用动态工具调度提升视频异常检测准确率
VTO: Visual Tool Orchestration for Video Anomaly Detection

- 引入过程监督强化学习,实现多步工具协同决策
- 在12类视觉工具上实现最高10.2%准确率提升
- 适合需要复杂推理的视频分析任务研究者
视频异常检测(VAD)因真实场景复杂多样而极具挑战性。传统深度学习方法泛化能力差,现有基于多模态智能体的方法依赖监督微调,难以应对复杂工具编排;标准强化学习则因奖励粒度粗导致过早终止。为此,我们提出VTO——一种过程监督强化学习框架。突破静态工具使用,VTO使智能体能动态探索环境。通过引入基础模型驱动的认知评估器,提供上下文感知语义反馈,并集成至过程监督认知对齐机制,实现细粒度、分步监督。通过显式惩罚逻辑截断、奖励完整因果链,优化多步推理策略。为支持该框架,我们构建了包含12类专用视觉工具的VAD-Tool层级工具集,涵盖目标跟踪到高危事件检测,并建立了用于多步推理评估的基准。在VAD-Tool上的实验表明,VTO显著优于基线,在工具调度上实现最高10.2%的绝对准确率提升。代码与数据见https://github.com/MICLAB-BUPT/VTO。
原文摘要 · Abstract (English)
Video anomaly detection (VAD) is a critical yet challenging task due to the complex and diverse nature of real-world scenarios. Traditional deep learning approaches are fundamentally limited by poor generalization across diverse scenarios. While multimodal agents offer a promising tool-learning paradigm for VAD, current systems relying on supervised fine-tuning struggle with complex orchestration, and standard reinforcement learning often causes premature termination due to coarse-grained outcome rewards. To address these challenges, we propose VTO, a process-supervised reinforcement learning framework. Moving beyond static tool usage, VTO enables the agent to dynamically explore and interact with the environment. Specifically, we introduce a foundation model-driven cognitive evaluator to provide context-aware semantic feedback, which is seamlessly integrated into a Process-Supervised Cognitive Alignment that delivers fine-grained, step-wise supervision. By explicitly penalizing logical truncation and rewarding complete causal chains, the agent optimizes its multi-step reasoning policy for interrelated tool orchestration. To support our proposed framework, we meticulously crafted VAD-Tool, a hierarchical visual tool set comprising 12 specialized vision tools spanning from entity tracking to high-stakes hazard detection, and established the corresponding benchmark for rigorous multi-step reasoning evaluation. Extensive experiments on VAD-Tool demonstrate that VTO significantly outperforms baselines, achieving up to a 10.2\% absolute accuracy improvement in tool scheduling. Code and data are available at https://github.com/MICLAB-BUPT/VTO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。