用语言智能代理高效找视频关键帧,提升问答准确率同时大幅降低计算量。
Agentic Keyframe Search for Video Question Answering
- 用语言代理驱动搜索算法,动态筛选视频关键帧
- 在EgoSchema上仅处理43.5%帧,准确率高1.8%
- 适合需要高效视觉推理的智能视频理解系统
视频问答(VideoQA)使机器能通过自然语言交互提取和理解视频信息,是实现智能的关键一步。然而,对视频的深度理解需求与高计算成本仍限制其广泛应用。为此,我们提出一种名为自适应关键帧搜索(AKeyS)的算法,可有效从冗余内容中识别关键信息。该方法将视频分段并组织为树状结构,利用语言代理动态评估启发式信息与移动代价,逐步扩展节点。当满足终止条件时,代理判断是否已收集足够关键帧并给出答案。在EgoSchema与NExT-QA数据集上的实验表明,AKeyS优于所有先前方法,在最高搜索效率下实现精准关键帧定位与有效视觉推理。例如,在EgoSchema子集上,准确率高出1.8%,且仅需处理43.5%的帧数,相比VideoTree显著降低计算开销。代码已公开于https://github.com/fansunqi/AKeyS。
原文摘要 · Abstract (English)
Video question answering (VideoQA) enables machines to extract and comprehend key information from videos through natural language interaction, which is a critical step towards achieving intelligence. However, the demand for a thorough understanding of videos and high computational costs still limit the widespread applications of VideoQA. To address it, we propose Agentic Keyframe Search (AKeyS), a simple yet powerful algorithm for identifying keyframes in the VideoQA task. It can effectively distinguish key information from redundant, irrelevant content by leveraging modern language agents to direct classical search algorithms. Specifically, we first segment the video and organize it as a tree structure. Then, AKeyS uses a language agent to estimate heuristics and movement costs while dynamically expanding nodes. Finally, the agent determines if sufficient keyframes have been collected based on termination conditions and provides answers. Extensive experiments on the EgoSchema and NExT-QA datasets show that AKeyS outperforms all previous methods with the highest keyframe searching efficiency, which means it can accurately identify key information and conduct effective visual reasoning with minimal computational overhead. For example, on the EgoSchema subset, it achieves 1.8% higher accuracy while processing only 43.5% of the frames compared to VideoTree. We believe that AKeyS represents a significant step towards building intelligent agents for video understanding. The code is publicly available at https://github.com/fansunqi/AKeyS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。