用工具调用分解查询，提升长视频关键帧检索精度
Decomposing Queries into Tool Calls for Long-Video Keyframe Retrieval
- 通过大模型将查询拆解为多个工具调用并指定合并逻辑
- 在关键帧检索任务中表现优异，尤其在字幕检索上领先5%
- 构建了带时间锚点的M2M基准，支持直接评估检索效果
关键帧选择是为长视频问答提供可验证视觉证据的直接方式。不同查询需求各异，找到正确帧取决于明确搜索目标。现有方法要么对每帧统一评分，要么将查询按固定模式分解后由单一视觉工具评估。我们提出ToolMerge：基于分解与融合的关键帧检索方法。一个基于大语言模型的规划器将查询分解为多个工具调用，并指定各工具排名如何通过布尔运算合并。为实现对检索结果的直接评估，我们构建了Molmo-2 Moments（M2M）基准，其中每个问题都通过构造方式锚定到特定时间区间。在问答、问题检索和字幕检索任务中，ToolMerge与现有方法相比具有竞争力，尤其在字幕检索上超出其他方法5%。代码与数据见https://github.com/michalsr/ToolMerge。
原文摘要 · Abstract (English)
Keyframe selection is a direct way to provide verifiable visual evidence for long-video question answering (QA). Queries differ in what they require, and finding the right frames depends on knowing what to look for. Existing keyframe selectors either score every frame against a single query, or decompose the query into a fixed schema evaluated by a single visual tool. We propose ToolMerge, a keyframe retrieval method based on decomposition and merging: an Large Language Model (LLM) based planner decomposes the query into tool calls and specifies how their per-tool rankings are merged using boolean operators. To evaluate retrieval directly, we construct Molmo-2 Moments (M2M), a benchmark in which every question is anchored to a specific time interval by construction. Across QA, question retrieval, and caption retrieval, ToolMerge is competitive with prior keyframe selectors, most notably on caption retrieval, outperforming other methods by 5%. Code and data can be found at https://github.com/michalsr/ToolMerge .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。