通过不确定性感知的思维链，提升大模型对长视频的理解可靠性。
VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoT
- 设计了可动态调整的思维链机制，适应长视频分析需求。
- 引入对语言模型与外部工具的不确定性估计，降低噪声影响。
- 在三个基准上平均领先13.1%，适合零样本长视频理解任务。
长视频理解在计算机视觉中日益重要但极具挑战。基于智能体的方法因能处理长序列并整合多种工具而受到关注，但仍存在两大问题：(1) 仅依赖大语言模型（LLM）的推理能力，缺乏针对长视频场景的专门增强机制；(2) 易受外部工具误差或噪声影响。为此，我们提出一种专用于长视频分析的链式思维（CoT）过程，其计划-调整模式支持LLM逐步规划并动态调整信息收集策略。同时，我们引入对LLM及外部工具的启发式不确定性估计，指导CoT流程。该机制使LLM能够评估新获取信息的可靠性，优化采集策略，并在综合答案时做出更稳健决策。实验证明，不确定性感知的CoT有效缓解了外部工具噪声，提升了输出可靠性。我们在VideoAgent2系统中实现该方法，包含通用上下文获取和专用工具设计等模块。在三个专用长视频基准及其子集上的评估显示，VideoAgent2相较之前最先进方法VideoAgent平均提升13.1%，在所有零样本方法中表现领先。
原文摘要 · Abstract (English)
Long video understanding has emerged as an increasingly important yet challenging task in computer vision. Agent-based approaches are gaining popularity for processing long videos, as they can handle extended sequences and integrate various tools to capture fine-grained information. However, existing methods still face several challenges: (1) they often rely solely on the reasoning ability of large language models (LLMs) without dedicated mechanisms to enhance reasoning in long video scenarios; and (2) they remain vulnerable to errors or noise from external tools. To address these issues, we propose a specialized chain-of-thought (CoT) process tailored for long video analysis. Our proposed CoT with plan-adjust mode enables the LLM to incrementally plan and adapt its information-gathering strategy. We further incorporate heuristic uncertainty estimation of both the LLM and external tools to guide the CoT process. This allows the LLM to assess the reliability of newly collected information, refine its collection strategy, and make more robust decisions when synthesizing final answers. Empirical experiments show that our uncertainty-aware CoT effectively mitigates noise from external tools, leading to more reliable outputs. We implement our approach in a system called VideoAgent2, which also includes additional modules such as general context acquisition and specialized tool design. Evaluation on three dedicated long video benchmarks (and their subsets) demonstrates that VideoAgent2 outperforms the previous state-of-the-art agent-based method, VideoAgent, by an average of 13.1% and achieves leading performance among all zero-shot approaches
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。