给大模型装上视频工具链，提升时空推理能力
Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
- 设计视频专用工具集，按时空顺序调度工具增强推理
- 在VideoMME和LongVideoBench上分别提升8.2%和4.6%
- 适合需要精准视频分析的智能助手研发人员
视频问答（VideoQA）是评估基础模型感知、理解与推理动态现实场景能力的关键任务。然而，现有多模态大模型在同时建模视频帧内空间关系与复杂时序因果演化方面仍存在困难。本文提出一个全面且可扩展的视频工具集，并设计时空推理框架（STAR），通过策略性调度时空工具，逐步定位视频关键区域。该框架在轻量级工具辅助下增强GPT-4o，在VideoMME上提升8.2%，在LongVideoBench上提升4.6%。代码已公开于https://github.com/fansunqi/VideoTool。
原文摘要 · Abstract (English)
Video Question Answering (VideoQA) task serves as a critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously modeling spatial relationships within video frames and understanding the causal dynamics of temporal evolution on complex and reasoning-intensive VideoQA task. In this work, we equip MLLM with a comprehensive and extensible Video Toolkit, to enhance MLLM's spatiotemporal reasoning capabilities and ensure the harmony between the quantity and diversity of tools. To better control the tool invocation sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. Our STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench. We believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants. The code is publicly available at https://github.com/fansunqi/VideoTool.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。