让视频智能体边推理边自我纠错，提升复杂视频理解能力。
CAViAR: Critic-Augmented Video Agentic Reasoning
- 智能体动态调用视频模块，根据前步结果决定下一步动作。
- 在LVBench、Neptune等数据集上超越已有方法，表现显著提升。
- 引入批评者机制，自动识别推理过程中的成功与失败路径。
近年来，视频理解在短片段感知任务上取得显著进展，但随着查询复杂度增加和视频时长延长，现有模型在需要复杂推理的任务上表现下降。本文探讨：能否利用现有感知能力实现更复杂的视频推理？我们构建了一个大型语言模型智能体，通过调用视频模块作为子代理或工具进行推理。不同于以往固定流程的方法（如Visual Programming、ViperGPT、MoReVQA），该智能体根据每一步模块调用的结果动态决策后续步骤。受文本推理领域启发，我们引入一个批评者模块，用于区分推理序列的成功与失败。实验表明，该智能体与批评者结合，在LVBench、Neptune、ActivityNet-RTL等多个基准上均取得优异表现。
原文摘要 · Abstract (English)
Video understanding has seen significant progress in recent years, with models' performance on perception from short clips continuing to rise. Yet, multiple recent benchmarks, such as LVBench, Neptune, and ActivityNet-RTL, show performance wanes for tasks requiring complex reasoning on videos as queries grow more complex and videos grow longer. In this work, we ask: can existing perception capabilities be leveraged to successfully perform more complex video reasoning? In particular, we develop a large language model agent given access to video modules as subagents or tools. Rather than following a fixed procedure to solve queries as in previous work such as Visual Programming, ViperGPT, and MoReVQA, the agent uses the results of each call to a module to determine subsequent steps. Inspired by work in the textual reasoning domain, we introduce a critic to distinguish between instances of successful and unsuccessful sequences from the agent. We show that the combination of our agent and critic achieve strong performance on the previously-mentioned datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。