用轻量模型实现一小时视频的精准问答与内容定位
FALCONEye: Finding Answers and Localizing Content in ONE-hour-long videos with multi-modal LLMs
- 结合VLM与LLM，用探索式搜索算法定位长视频内容
- 在FALCON-Bench上超越所有7B开源模型，支持答案+时间窗口返回
- 推理成本降十倍，适用于长视频问答与多任务泛化
在长达一小时的视频中查找信息对顶尖视觉语言模型（VLM）仍是难题，因视觉内容编码常超出上下文窗口限制。为此，我们提出FALCONEye，一种基于无训练、模型无关元架构的视频智能体，由一个VLM和一个轻量级LLM组成。该系统通过受校准置信度引导的探索式搜索算法回答开放性问题。我们还构建了FALCON-Bench基准，将视频问答扩展为需要返回答案及其支持的时间窗口的任务。仅使用7B VLM与轻量级LLM，FALCONEye在FALCON-Bench上超越所有开源7B VLM及同类代理。它在更短视频的MLVU基准上也展现出良好泛化能力，单细节任务表现优于GPT-4o，同时推理成本降低约一个数量级。
原文摘要 · Abstract (English)
Finding information in hour-long videos is a challenging task even for top-performing Vision Language Models (VLMs), as encoding visual content quickly exceeds available context windows. To tackle this challenge, we present FALCONEye, a novel video agent based on a training-free, model-agnostic meta-architecture composed of a VLM and a Large Language Model (LLM). FALCONEye answers open-ended questions using an exploration-based search algorithm guided by calibrated confidence from the VLM's answers. We also introduce the FALCON-Bench benchmark, extending Question Answering problem to Video Answer Search-requiring models to return both the answer and its supporting temporal window for open-ended questions in hour-long videos. With just a 7B VLM and a lightweight LLM, FALCONEye outscores all open-source 7B VLMs and comparable agents in FALCON-Bench. It further demonstrates its generalization capability in MLVU benchmark with shorter videos and different tasks, surpassing GPT-4o on single-detail tasks while slashing inference cost by roughly an order of magnitude.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。